Search

The generative adversarial network (GAN) is successfully applied to study the perceptual single image super-resolution (SISR). However, since the GAN is data-driven, it has a fundamental limitation on restoring real high frequency information for an unknown instance (or image) during test. On the other hand, the conventional model-based methods have a superiority to achieve instance adaptation as they operate by considering the statistics of each instance (or image) only. Motivated by this, we propose a novel model-based algorithm, which can extract the detail layer of an image efficiently. The detail layer represents the high frequency information of image and it is constituted of image edges and fine textures. It is seamlessly incorporated into the GAN and serves as a prior knowledge to assist the GAN in generating more realistic details. The proposed method, named DSRGAN, takes advantages from both the model-based conventional algorithm and the data-driven deep learning network. Experimental results demonstrate that the DSRGAN outperforms the state-of-the-art SISR methods on perceptual metrics, meanwhile achieving comparable results in terms of fidelity metrics. Following the DSRGAN, it is feasible to incorporate other conventional image processing algorithms into a deep learning network to form a model-based deep SISR. [ABSTRACT FROM AUTHOR]

Blind image deblurring is a long standing challenging problem in image processing and low-level vision. Recently, sophisticated priors such as dark channel prior, extreme channel prior, and local maximum gradient prior, have shown promising effectiveness. However, these methods are computationally expensive. Meanwhile, since these priors involved subproblems cannot be solved explicitly, approximate solution is commonly used, which limits the best exploitation of their capability. To address these problems, this work firstly proposes a simplified sparsity prior of local minimal pixels, namely patch-wise minimal pixels (PMP). The PMP of clear images is much more sparse than that of blurred ones, and hence is very effective in discriminating between clear and blurred images. Then, a novel algorithm is designed to efficiently exploit the sparsity of PMP in deblurring. The new algorithm flexibly imposes sparsity inducing on the PMP under the maximum a posterior (MAP) framework rather than directly uses the half quadratic splitting algorithm. By this, it avoids non-rigorous approximation solution in existing algorithms, while being much more computationally efficient. Extensive experiments demonstrate that the proposed algorithm can achieve better practical stability compared with state-of-the-arts. In terms of deblurring quality, robustness and computational efficiency, the new algorithm is superior to state-of-the-arts. Code for reproducing the results of the new method is available at https://github.com/FWen/deblur-pmp.git. [ABSTRACT FROM AUTHOR]

It is challenging to extend the support region of state-of-the-art local edge-preserving filtering approaches to the entire input image on account of huge memory cost and heavy computational burden. In this paper, we propose an O(N) time recursive non-local edge-aware filter. A novel graph and a linear time algorithm are presented to effectively propagate information across the entire image. In this graph, information is propagated along all directions to avoid visual artifacts. Both the intensity similarity and the spatial affinity are utilized to estimate the similarity of neighboring pixels, and the distance of any two pixels is the length of the path between these two pixels on a binary tree. Specifically, the input image is filtered at four directions, namely left-to-right, right-to-left, top-to-bottom and bottom-to-top. In each pass, the un-normalized output and the normalization constant of the root node are computed recursively from leaf nodes to the root node on a binary tree in linear time. The filtering output is the average of the outputs for these four directions. A comparison with other non-local edge-aware filters is presented to show the advantages of our approach. Extensive experiments demonstrate that our recursive non-local edge-preserving filter is effective in a variety of computer vision and image processing applications, including edge-aware filtering, detail enhancement, stylization, and stereo matching. [ABSTRACT FROM AUTHOR]

Being able to predict the salient object is of fundamental importance in image processing and computer vision. With numerous approaches proposed for automatic image and video salient object detection, much less work has been dedicated to detecting and segmenting salient objects from light fields. In this article, based on the intrinsic characteristics of light fields, we carefully explore the complementary coherence among multiple cues including spatial, edge and depth information, and elaborately design a multi-task collaborative network for light field salient object detection. More specifically, the correlation mechanisms among edge detection, depth inference and salient object detection are carefully investigated to facilitate the representative saliency features. We first model the coherence among low-level features and heuristic semantic priors, as well as the edge information. Subsequently, the depth-oriented saliency features are derived from the geometry of light fields, in which the 3D convolution operation is leveraged with powerful representation capability to model the disparity correlations among multiple viewpoint images. Finally, a feature-enhanced salient object generator is developed to integrate these complementary saliency features, leading to the final salient object predictions for light fields. Quantitative and qualitative experiments demonstrate the superiority of our proposed model against the state-of-the-art methods over the public light field salient object detection datasets. [ABSTRACT FROM AUTHOR]

A reliable and accurate visual object 6DoF pose tracking system for texture-less objects plays an important role in various fields of modern industry hence it has been a hot research topic for decades. Traditional feature-based pose tracking methods require rich feature points on objects, and it cannot handle texture-less objects. To tackle this problem, the paper proposes a novel edge-based method for continuous 6DOF pose tracking of texture-less objects. The pose of the object is estimated by minimizing the matching error between extracted image edges and re-projected CAD model edges. The matching error is represented using the Directional Chamfer Matching (DCM) Tensor. Compared with previous methods, the proposed system improves the overall performance of pose tracking system in two ways. Firstly, the method proposes an analytical mathematic model in the optimization process; Secondly, the method proposes an adaptive edge point weighting algorithm to tackle the occlusion or edge weakness problem. Both methods help improve the accuracy and robustness of the pose estimation system. With the benefit of GPU acceleration on DCM Tensor calculation the proposed method could run in real time. Extensive experiments are conducted both on public datasets and CG-rendered synthetic datasets to validate the accuracy and robustness of the proposed algorithm against other state-of-the-art methods. [ABSTRACT FROM AUTHOR]

Blur is one of the most common distortions that degrade natural images. This stimulates the blossom of sharpness assessment metrics. Existing sharpness metrics possess good performance for evaluating simulated blur, but are limited for the more common realistic blur that are introduced during image capture and processing in real life. To this end, we propose an effective Realistic Blur Assessment method (RBA) based on discrepancy learning. First, motivated by the fact that the distortion-free reference images are usually unavailable in practice, but the Human Visual System (HVS) can still accurately perceive image sharpness by quantifying the perceptual discrepancy between the distorted image and the hallucinated reference image in mind, we propose to train a discrepancy generation model to automatically generate the discrepancy map from the distorted image analogous to the HVS. This is achieved by using a deep neural network with rich training images. With the discrepancy map, two sharpness-aware features, i.e. sparse representation based entropy of primitive and content-guided variation of power, are then extracted to severally quantify spatial visual information amount and spectral power. Finally, the two features are integrated to produce the overall sharpness score. Extensive experiments demonstrate the superiority of the proposed method over the state-of-the-arts. [ABSTRACT FROM AUTHOR]

Wavelet transform using Haar filter is a fast process for decomposing an image into low and high frequency sub-bands, which is an important step in multi-focus image fusion. This transformation alone is usually not sufficient to produce satisfying fused images, and therefore additional tools like focused region decision map and feature extraction are needed to enhance the fusion quality. In this paper, we propose a multi-focus image fusion algorithm that is suitable for efficient hardware implementation. The algorithm speed is significantly improved by utilizing Haar wavelet and computationally simple fusion rules. To further improve the circuit power and delay, we limit the logic operations to additions, subtractions, and multiplications only. Experiments on different benchmarks demonstrate that our proposed algorithm can reduce the fusion time by up to 99.97% compared with the state-of-the-art competing methods without compromising the fusion quality. In addition, the hardware implementation of the proposed algorithm reduces the circuit area and delay by 79.5% and 92.5%, respectively, when compared to the most competitive algorithm in this paper while maintaining similar fusion quality. [ABSTRACT FROM AUTHOR]

Vehicle detection in aerial images is a crucial image processing step for many applications such as screening of large areas as used for surveillance, reconnaissance, or rescue tasks. In recent years, several deep learning-based frameworks have been proposed for object detection. However, these detectors were developed for data sets that considerably differ from aerial images. In this paper, we systematically investigate the potential of fast R-CNN and faster R-CNN for aerial images, which achieve top performing results on common detection benchmark data sets. Therefore, the applicability of eight state-of-the-art object proposal methods used to generate a set of candidate regions and of both detectors is examined. Relevant adaptations to account for the characteristics of the aerial images are provided. To overcome the shortcomings of the original approach in the case of handling small instances, we further propose our own networks that clearly outperform state-of-the-art methods for vehicle detection in aerial images. Furthermore, we analyze the impact of the different adaptations with respect to various ground sampling distances to provide a guideline for detecting small objects in aerial images. All experiments are performed on two publicly available data sets to account for differing characteristics such as varying object sizes, number of objects per image, and varying backgrounds. [ABSTRACT FROM AUTHOR]

Guided image filtering has been applied widely in recent years as a solution to the ever-increasing demand of high-performance filtering, especially for real-time image/video processing. The lately proposed gradient domain guided image filter (GDGIF) is one of the typical works focusing on improving the quality of the filtering result of the original guided image filter (GIF), dealing with the halo-artifacts problem for edge-preserving smoothing. However, due to the involvement of global pixel values in the computation, high computation complexity, and additional complex operation, there is no existing VLSI design for the gradient guided filter. This paper presents a high-throughput VLSI architecture for real-time full-HD ($1920\times 1080$) GDGIF. Four techniques are used in the newly proposed design to reduce the computation complexity and increase the processing throughput. First, a pre-processing stage is introduced to address the problem of global parameters required in the GDGIF. Second, multi-scale down-samplings are adopted for reducing circuit size and processing time. Then, a parallel structure operated at a higher frequency is used at the output stage to restore the full image size and to achieve high throughput at the same time. At last, approximated computation is introduced to eliminate speed bottleneck by simplifying the complex exponentiation operation. Based on the STM 65-nm CMOS technology, the implementation results show that the proposed architecture is able to support full-HD image filtering at a throughput above 75 frame/s, with a design area of 586314 $\mu \text{m}^{2}$ and a power consumption of 18.5 mW. [ABSTRACT FROM AUTHOR]

Despite recent advances made in salient object detection using graph theory, the approach still suffers from accuracy problems when the image is characterized by a complex structure, either in the foreground or background, causing erroneous saliency segmentation. This fundamental challenge is mainly attributed to the fact that most existing graph-based methods take only the adjacently spatial consistency among graph nodes into consideration. In this paper, we tackle this issue from a coarse-to-fine perspective and propose a two-stage-graphs approach for salient object detection, in which two graphs having the same nodes but different edges are employed. Specifically, a weighted joint robust sparse representation model, rather than the commonly used manifold ranking model, helps to compute the saliency value of each node in the first-stage graph, thereby providing a saliency map at the coarse level. In the second-stage graph, along with the adjacently spatial consistency, a new regionally spatial consistency among graph nodes is considered in order to refine the coarse saliency map, assuring uniform saliency assignment even in complex scenes. Particularly, the second stage is generic enough to be integrated in existing salient object detectors, enabling improved performance. Experimental results on benchmark data sets validate the effectiveness and superiority of the proposed scheme over related state-of-the-art methods. [ABSTRACT FROM AUTHOR]

Achieving good text detection and recognition results for multi-script-oriented images is a challenging task. First, we explore bit plane slicing in order to utilize the advantage of the most significant bit information to identify text components. A new iterative nearest neighbor symmetry is then proposed based on shapes of convex and concave deficiencies of text components in bit planes to identify candidate planes. Further, we introduce a new concept called mutual nearest neighbor pair components based on gradient direction to identify representative pairs of texts in each candidate bit plane. The representative pairs are used to restore words with the help of edge image of the input one, which results in text detection results (words). Second, we propose a new idea by fixing window for character components of arbitrary oriented words based on angular relationship between sub-bands and a fused band. For each window, we extract features in contourlet wavelet domain to detect characters with the help of an SVM classifier. Further, we propose to explore HMM for recognizing characters and words of any orientation using the same feature vector. The proposed method is evaluated on standard databases such as ICDAR, YVT video, ICDAR, SVT, MSRA scene data, ICDAR born digital data, and multi-lingual data to show its superiority to the state of the art methods. [ABSTRACT FROM AUTHOR]

The active contour model is a widely used technique for automatic object contour extraction. Existing methods based on this model can perform with high accuracy, even in the case of complex contours, but challenging issues remain, like the need for precise contour initialization for high curvature boundary segments or the handling of cluttered backgrounds. To deal with such issues, this paper presents a salient object extraction method, the first step of which is the introduction of an improved edge map that incorporates edge direction as a feature. The direction information in the small neighborhoods of image feature points is extracted, and the images’ prominent orientations are defined for direction-selective edge extraction. Using such improved edge information, we provide a highly accurate shape contour representation, which we also combine with texture features. The principle of the paper is to interpret an object as the fusion of its components: its extracted contour and its inner texture. Our goal in fusing textural and structural information is twofold: it is applied for automatic contour initialization, and it is also used to establish an improved external force field. This fusion then produces highly accurate salient object extractions. We performed extensive evaluations, which confirm that the presented object extraction method outperforms parametric active contour models and achieves higher efficiency than the majority of the evaluated automatic saliency methods. [ABSTRACT FROM AUTHOR]

Frame rate up-conversion (FRUC) is a technology for generating a higher frame rate video from one with a lower frame rate. The multi-view plus depth map (MVD) system has been recently considered a promising solution for three-dimensional video systems. Accordingly, an FRUC efficient solution for the MVD system is greatly needed. In this paper, we thus propose an FRUC approach that considers the MVD system characteristics. It divides a frame into multiple layers using a depth map, and it reconstructs occluded regions using neighboring views and depth map information. With reconstructed information, it accurately performs motion estimation when a region occlusion occurs. Experimental results demonstrated that the proposed method significantly improved the quality of interpolated frames, and it can effectively complement other conventional FRUC approaches. [ABSTRACT FROM AUTHOR]

Objectness has recently become a standard step in many computer vision tasks. Among various techniques, those based on hierarchical image segmentation play a fundamental role for developments in new data modalities. In this paper, we address the problem of objectness in RGB-D images and propose a novel and effective approach, namely, hierarchical image segmentation ensemble (HISE). Different from existing image segmentation based methods that generate object segments or proposals largely by heuristics or empirical rules, HISE learns superpixel mergings with a hierarchical tree-structured ensemble, where individual merging models of the ensemble are formed by traversing different paths of the tree, and where both the merging accuracy and proposal diversity are emphasized. Furthermore, we use efficient feature measurements that support easy integration of additional clues. Extensive experiments conducted on the benchmark NYU-v2 RGB-D and SUN RGB-D data sets show the competency of our proposed method. [ABSTRACT FROM AUTHOR]

Contrast enhancement is a crucial image processing step for image quality control. However, images enhanced by conventional contrast enhancement methods can have negative effects on the performance of image quality control. The most commonly observed effects are over- and under-enhancement effects on images, which cause significant loss of fine textures in images. This paper developed a new contrast enhancement algorithm based on an entropy-preserving mapping prior that improves on conventional contrast enhancement methods. By creating a closed-form solution for enhancing the image contrast under this novel prior and learning the coefficients of the solution using an unsupervised learning strategy, an image’s contrast and texture can be effectively recovered. The experimental results verify that our proposed method clearly outperforms the existing state-of-the-art methods in terms of both quantitative estimation and qualitative human visual inspection. [ABSTRACT FROM AUTHOR]

In this paper, a novel image quality assessment (IQA) model for the screen content images (SCIs) is proposed by using multi-scale difference of Gaussian (MDOG). Motivated by the observation that the human visual system (HVS) is sensitive to the edges while the image details can be better explored in different scales, the proposed model exploits MDOG to effectively characterize the edge information of the reference and distorted SCIs at two different scales, respectively. Then, the degree of edge similarity is measured in terms of the smaller-scale edge map. Finally, the edge strength computed based on the larger-scale edge map is used as the weighting factor to generate the final SCI quality score. Experimental results have shown that the proposed IQA model for the SCIs produces high consistency with human perception of the SCI quality and outperforms the state-of-the-art quality models. [ABSTRACT FROM AUTHOR]

Due to the rapid growth of technology and the dropping cost of cameras, multiview imaging applications have attracted many researchers in recent years. Free viewpoint and 3D Televisions are among these interesting applications. One of the problems that should be solved to realize such applications is rendering. In this paper, we propose an optical flow-assisted adaptive patch-based view synthesis algorithm. This patch-based scheme reduces the size and number of holes during reconstruction. The size of patch is determined in response to edge information for better reconstruction, especially near the boundaries. In the first stage of the algorithm, disparity is obtained using optical flow estimation. Then, a reconstructed version of the left and right views is generated using our adaptive patch-based algorithm. The mismatches between each view and its reconstructed version are obtained in the mismatch detection steps. This stage results in two masks as outputs, which help with the refinement of disparities and the selection of the best patches for final synthesis. Finally, the remaining holes are filled using our simple hole-filling scheme and the refined disparities. The objective and subjective performances of the proposed algorithm are compared with recent methods. The results show that the proposed algorithm achieves an improvement of 2.14 dB on average. [ABSTRACT FROM AUTHOR]

The screen content images (SCIs) quality influences the user experience and the interactive performance of remote computing systems. With numerous approaches proposed to evaluate the quality of natural images, much less work has been dedicated to reduced-reference image quality assessment (RR-IQA) of SCIs. Here, we propose an RR-IQA method from the perspective of SCI visual perception. In particular, the quality of the distorted SCI is evaluated by comparing a set of extracted statistical features that consider both primary visual information and unpredictable uncertainty. A unique property that differentiates the proposed method from previous RR-IQA methods for natural images is the consideration of behaviors when human subjects view the screen content, which motivates us to establish the perceptual model according to the distinct properties of SCIs. Validations based on the screen content IQA database show that the proposed algorithm provides accurate predictions across a wide range of SCI distortions with negligible transmission overhead. [ABSTRACT FROM PUBLISHER]

In this paper, we present a frame rate upconversion method using optical flow motion estimation and a patch-based reconstruction scheme. First, forward and backward motion vectors (MVs) are obtained using an optical flow algorithm, and reconstructed versions of the current and previous frames are generated by our patch-based reconstruction scheme. Using the original and reconstructed versions of the current and previous frames, two mismatch masks are obtained. Then, two versions of the middle frame are generated using a patch-based scheme, with estimated MVs and the current and previous frames. Finally, a middle mask, which identifies the mismatch areas of the two middle frames, is reconstructed. Using these three masks, the best candidates for interpolation are selected and fused to obtain the final middle frame. Due to the patch-based nature of our reconstruction scheme, most of the holes and cracks will be filled. Although there is always a probability of having holes, the size and number of such holes are much smaller than those that would be generated using pixel-based mapping. The rare holes are filled using existing hole-filling algorithms. The experimental results and a comparison of our method with existing algorithms show that our method performs better in terms of both objective and subjective quality of the final interpolated frames. The average peak signal-to-noise ratio (PSNR) improvement of our method is 1–2 dB. [ABSTRACT FROM AUTHOR]

High-quality and cost-efficient image upscaling design is very important for many real-time video processing applications, especially when the display panel resolution reaches ultrahigh definition. Compared with New Edge-Directed Interpolation (NEDI) based implicit edge directional upscaling, explicit methods require less computational resource and more easily reach real-time performance, especially when the required image definition and upscaling ratio are very high. Nevertheless, the investigation of applications of explicit methods in video processing systems remains largely missing arguably because it is commonly believed that explicit edge-directed interpolation tends to introduce unexpected artifacts because of inaccurate detection and hence its image quality is relatively poor. This paper proposes an explicit edge-directed adaptive interpolation method that leverages more sophisticated edge detection and orientation estimation algorithms to avoid misinterpolation, thereby providing similar or even better image quality than those with implicit methods. Targeting the real-time 4K video display system, the proposed edge-directed image upscaling algorithm is further implemented with an efficient very-large-scale integration (VLSI) architecture. The experimental results demonstrate that the proposed interpolation algorithm outperforms previous explicit and implicit edge-directed methods in both objective and subjective tests. The presented VLSI implementation further demonstrates that the maximum output video sequence of the proposed interpolation method can reach 4\mathrm k\times 2\textk @60 Hz with a reasonable hardware cost. [ABSTRACT FROM AUTHOR]

Screen content images are originally captured in a full-chroma format. The chroma downsampling, which is commonly applied to the chroma component in screen content image representation and processing (e.g., YUV4:2:0 compression), will significantly degrade the image quality and create annoying artifacts such as blur and color shifting. To tackle this problem, in this paper we propose luma aware chroma downsampling and upsampling algorithms to jointly improve the quality of the chroma image reconstruction. Guided by the luma information, the chroma upsampling algorithm is proposed with the utilization of major color and index map representation. The geometric information-based linear mapping is developed to transfer the structure of luma to the interpolated chroma. Subsequently, the error sensitivity of the upsampling method is analyzed, and content dependent downsampling algorithm is presented to minimize the error sensitivity function. We further explore the applicability of the proposed scheme in the scenario of screen content compression, targeting at improving the decoded chroma image quality for display. Extensive experimental results demonstrate the viability and efficiency of the proposed scheme. [ABSTRACT FROM AUTHOR]

Stereo matching is a challenging problem, and high-accuracy stereo matching is still required in various computer vision applications, e.g., 3-D scanning, autonomous navigation, and 3-D reconstruction. Therefore, we present a novel image-guided stereo matching algorithm, which employs the efficient combined matching cost and multistep disparity refinement, to improve the accuracy of existing local stereo matching algorithms. Different from all the other methods, we introduce a guidance image for the whole algorithm. This filter-based guidance image is generated by extracting the enhanced information from the raw stereo image. The combined matching cost consists of the novel double-RGB gradient, the improved lightweight census transform, and the image color. This cost measurement is robust against image noise and textureless regions in computing the matching cost. Furthermore, a new systemic multistep refinement process, which includes outlier classification, four-direction propagation, leftmost propagation, and an exponential step filter, is proposed to remove the outliers in the raw disparity map. Experiments on the Middlebury benchmark demonstrate our algorithm’s superior performance that it ranks first among the 158 submitted algorithms. Moreover, the proposed method is also robust on the 30 Middlebury data sets and the real-world Karlsruhe Institute of Technology and Toyota Technological Institute benchmark. [ABSTRACT FROM AUTHOR]

Binary image descriptors, which derive image feature description from the local image patches directly, are widely adopted in the mobile and embedded applications due to lower computational complexity and memory requirement. With the aim of improving the computation efficiency without degrading recognition performance, a lightweight binary robust descriptor is proposed based on the analysis of the state-of-the-art binary descriptors in this paper. A directional edge detection and optimized keypoint score function are developed to refine the keypoints. In addition, rotation invariance is achieved by executing circular symmetric-based descriptor generation and a coarse-grained orientation calculation method concurrently. The experimental results demonstrate that the proposed keypoint detector and binary descriptor achieve more than two times speedup and at least 23.6% improvement in processing speed with comparable performance, respectively. Furthermore, a very large scale integration architecture is also designed based on in-depth exploration of bit-level and task-level parallelism. Based on the postlayout simulation in a TSMC 65-nm CMOS process, the accelerator can achieve 135 frames/s on 1080p image while only consuming 87.5 mW at a 200-MHz operating frequency. [ABSTRACT FROM PUBLISHER]

A novel superpixel extraction algorithm using a higher order energy optimization framework is proposed in this paper. We first adopt the $k$ -means clustering technique to quickly get an initial superpixel result. Then a higher order energy function is employed to optimize and refine these initial superpixels. We use a more general higher order energy function that includes a first-order data term, a second-order smoothness term, and a higher order term. The presegments are employed to provide the prior information of sufficient edges and segment regions for our higher order energy term. According to the texture measurement in different local regions, our algorithm adaptively computes the proper ratios of different energy terms to obtain a better superpixel performance. The experimental results demonstrate that our method using the higher order energy generates better results with well-aligned boundaries and homogeneous effects than the existing superpixel algorithms. [ABSTRACT FROM PUBLISHER]

The locality sensitive histogram (LSH) injects spatial information into the local histogram in an efficient manner, and has been demonstrated to be very effective for visual tracking. In this paper, we explore the application of this efficient histogram in two important problems. We first extend the LSH to linear time bilateral filtering, and then propose a new type of histogram for efficiently computing edge-preserving nearest neighbor fields (NNFs). While the existing histogram-based bilateral filtering methods are the state of the art for efficient grayscale image processing, they are limited to box spatial filter kernels only. In our first application, we address this limitation by expressing the bilateral filter as a simple ratio of linear functions of the LSH, which is able to extend the box spatial kernel to an exponential kernel. The computational complexity of the proposed bilateral filter is linear in the number of image pixels. In our second application, we derive a new bilateral weighted histogram (BWH) for NNF. The new histogram maintains the efficiency of LSH, which allows approximate NNF to be computed independent of patch size. In addition, BWH takes both spatial and color information into account, and thus provides higher accuracy for histogram-based matching, especially around color edges. [ABSTRACT FROM PUBLISHER]

In this paper, we propose a four-direction residual interpolation (FDRI) method for color filter array interpolation. The proposed algorithm exploits a guided filtering process to generate the tentative image. The residual image is generated by exploiting the tentative and original images. We use an FDRI algorithm to more accurately estimate the missing pixel values; the estimated image is adaptively combined with a joint inverse gradient weight. Based on the experimental results, the proposed method provides a superior performance in terms of objective and subjective quality compared with the conventional state-of-the-art demosaicking methods. [ABSTRACT FROM PUBLISHER]

We develop a framework for estimating the noise level of a natural image using two important statistics: 1) high kurtosis and 2) scale invariance in transform domain. By exploring the said priors of natural image statistics in 2-D discrete cosine transform (DCT) domain, we reveal the limitations of these statistics for images with highly directional edges or large smooth areas. Then, we derive a novel two-step estimation scheme for noise variance: 1) in preliminary estimation, an integration of wavelet and nondirectional DCT transform is used to alleviate the influence of image’s structures and 2) a noise-injection rectification is further devised to deal with the noise-free image contents. A simulation and comparative study demonstrates that this algorithm reliably infers noise variance and its robustness over wide ranges of visual content and noise levels, while outperforming some relevant methods. This paper can significantly improve the performance of existing denoising techniques that require the noise variance as a critical parameter. [ABSTRACT FROM PUBLISHER]

Patch-based denoising, while effective, requires expensive pairwise patch comparisons. We present a novel fast patch-based denoising technique based on patch geodesic paths (PatchGPs). PatchGPs treat image patches as nodes and patch differences as edge weights for computing the shortest (geodesic) paths. The distance defined by the PatchGP can then be used as a similarity metric for image denoising. We first show that, for natural images, PatchGPs can be approximated by minimum hop paths (MHPs) that correspond to Euclidean line paths connecting two patch nodes. The denoising kernel is constructed using patches along discretized MHP search directions. We apply a weight propagation scheme to robustly and efficiently compute the path distance for each MHP. Our technique handles noise at multiple scales by analyzing the noise distribution (through wavelet decomposition) at each scale. Experiments show that our approach maintains the high quality of patch-based denoising but is a few orders of magnitude faster. We also demonstrate how PatchGP can be used for fast Bayer pattern (raw) denoising and image detail enhancement. [ABSTRACT FROM PUBLISHER]

The autoregressive (AR) model has been widely used in signal processing for its effective estimation, especially in image processing. Many dedicated $2\times $ interpolation algorithms adopt the AR model to describe the strong correlation between low-resolution (LR) pixels and high-resolution (HR) pixels. However, these AR model-based methods closely depend on the fixed relative position between LR pixels and HR pixels that are nonexistent in the general scale interpolation. In this paper, we present an adaptive general scale interpolation algorithm that is capable of arbitrary scaling factors considering the nonstationarity of natural images. Different from other dedicated $2\times $ interpolation methods, the proposed AR terms are modeled by pixels with their adjacent unknown HR neighbors. To compensate for the information loss caused by mismatches of AR models, we consider a weighting scheme suitable for general scale situations based on the pixel similarity to increase accuracy of the estimation. Comprehensive experiments demonstrate the effectiveness of the proposed method on general scaling factors. The maximum gain of peak signal-to-noise ratio is 2.07 dB compared with segment adaptive gradient angle in $1.5 \times $ enlargements. To evaluate the performance in resolution adaptive video coding, we have also tested our method on Joint Scalable Video Model codec and obtained better subjective quality and rate-distortion performance. [ABSTRACT FROM PUBLISHER]

This paper addresses high performance depth coding in 3D video by making good use of its coded texture video counterpart. The relationship between the depth and its associated texture video in terms of coding mode and motion vector is carefully examined. Our statistical study suggests that the skip-coding mode and its associated motion vectors in the coded texture can be shared for depth coding by saving bit rate at the cost of little increase of distortion, which subsequently results in a nonsequential coding of the depth map. In this sense, coding/prediction of a block can be performed using the skip-coded blocks below and right, which are not available in the conventional sequential coding, thus producing the so-called omnidirectional blocks predicted in the intra-coding by making the best use of (at most) four neighboring blocks. Moreover, in view of the depth-texture structure similarity, a depth-texture cooperative clustering-based prediction method is proposed for cluster-based depth prediction in the intra-coding, which exploits the structure similarity for the current coding block and its neighboring pixels around the block. On the other hand, some large prediction errors may be present for the depth-texture misaligned pixels, which may greatly compromise the coding performance. To deal with these large residuals induced by the depth-texture misalignment, a simple yet effective detection and rectification approach is incorporated in the proposed depth coding scheme. Experimental results show that our proposed depth coding scheme achieves superior rate-distortion performance compared with other relevant coding methods. [ABSTRACT FROM PUBLISHER]

This paper investigates a general framework to discover categories of unlabeled scene images according to their appearances (i.e., textures and structures). We jointly solve the two coupled tasks in an unsupervised manner: 1) classifying images without predetermining the number of categories and 2) pursuing generative model for each category. In our method, each image is represented by two types of image descriptors that are effective to capture image appearances from different aspects. By treating each image as a graph vertex, we build up a graph and pose the image categorization as a graph partition process. Specifically, a partitioned subgraph can be regarded as a category of scenes and we define the probabilistic model of graph partition by accumulating the generative models of all separated categories. For efficient inference with the graph, we employ a stochastic cluster sampling algorithm, which is designed based on the Metropolis–Hasting mechanism. During the iterations of inference, the model of each category is analytically updated by a generative learning algorithm. In the experiments, our approach is validated on several challenging databases, and it outperforms other popular state-of-the-art methods. The implementation details and empirical analysis are presented as well. [ABSTRACT FROM PUBLISHER]

In this paper, we propose a novel energy optimization framework to accurately estimate surface normal and reflectance of an object from an input image sequence. Input images are captured from a fixed viewpoint under varying lighting conditions. In the proposed approach we combine photometric stereo and Retinex constraints into our energy function. To formulate inter-image constraints, shading information is added to the Lambertian model to account for shadows. For intra-image constraints, we moderate the strength of shading smoothness according to shadow mask and normal variations. By minimizing this energy function we are able to recover accurate surface normals and reflectance. Experimental results show that our approach yields more realistic normal map and accurate albedo map than the state-of-the-art uncalibrated photometric stereo algorithms. As for intrinsic image decomposition, results on the real and synthetic scenes show that the proposed approach outperforms previous ones. [ABSTRACT FROM PUBLISHER]

The challenge of single-image dehazing mainly comes from double uncertainty of scene radiance and scene transmission. Most existing methods focus on restoring the visibility of hazy images and tend to derive a rough estimate of scene transmission. Unlike previous work, in this paper we advocate the significance of accurate transmission estimation and recast our problem as deriving the optimal transmission map directly from the haze model under two scene priors. We introduce theoretic and heuristic bounds of scene transmission to guide the optimum and show that the proposed theoretic bound happens to justify the well-known dark channel prior of haze-free images. With the constraints on the solution space, we then incorporate two scene priors, including locally consistent scene radiance and context-aware scene transmission, to formulate a constrained minimization problem and solve it by quadratic programming. The global optimality is guaranteed. Simulations on synthetic data set quantitatively verify the accuracy and show that the transmission map successfully captures fine-grained depth boundaries. Experimental results on color/gray-level images demonstrate that our method outperforms most state of the arts in terms of both accurate transmission maps and realistic haze-free images. [ABSTRACT FROM AUTHOR]

This paper presents a maximum a posteriori (MAP) framework to incorporate a spatial consistency prior model for depth reconstruction in the shape-from-focus (SFF) process. Existing SFF techniques, which reconstruct a dense 3-D depth from multifocus image frames, usually have poor performance over low-contrast regions and usually need a large number of frames to achieve satisfactory results. To overcome these problems, a new depth reconstruction process is proposed to estimate the depth values by solving an MAP estimation problem with the inclusion of a spatial consistency model. This consistency model assumes that within a local region, the depth value of each pixel can be roughly predicted by an affine transformation of the image features at that pixel. A local learning process is proposed to construct the consistency model directly from the multifocus image sequence. By adopting this model, the depth values can be inferred in a more robust way, especially over low-contrast regions. In addition, to improve the computational efficiency, a cell-based version of the MAP framework is proposed. Experimental results demonstrate the effective improvement in accuracy and robustness as compared with existing approaches over real and synthesized image data. In addition, experimental results also demonstrate that the proposed method can achieve quite impressive performance, even with only the use of a few image frames. [ABSTRACT FROM AUTHOR]

A very-large-scale integration capable of extracting motion features from moving images in real time has been developed employing row-parallel and pixel-parallel architectures based on the digital pixel sensor technology. Directional edge filtering of input images is carried out in row-parallel processing to minimize the chip real estate. To achieve a real-time response of the system, a fully pixel-parallel architecture has been explored in adaptive binarization of filtered images for essential feature extraction as well as in their temporal integration and derivative operations. As a result, self-speed-adaptive motion feature extraction has been established. The chip was designed and fabricated in a 65-nm CMOS technology and used to build an object detection system. Motion-sensitive target image localization was demonstrated as an illustrative example. [ABSTRACT FROM AUTHOR]

Image matting refers to the problem of foreground extraction from an image and transparency determination of the pixels. Although other matting algorithms have been proposed, most are not sufficiently robust to obtain satisfactory matting results in different regions of an image, such as smooth regions, nonuniform color distribution regions and isolated color regions. This paper proposes a novel matting algorithm that can extract high-quality mattes from different regions of an image. Our proposed algorithm combines propagation and color-sampling methods. Unlike previous propagation-based approaches that use either local or nonlocal propagation methods, our propagation framework adaptively uses both local and nonlocal processes according to the detection results of the different regions in the image. Our color-sampling strategy, which is based on the characteristics of the superpixel, uses a simple sample selection criterion and requires significantly less computational cost than previous color-sampling methods. Experimental results show that our adaptive propagation framework, alone, outperforms the state-of-the-art propagation-based approaches. Combined with our color-sampling method, it can effectively handle different regions in the image and produce both visually and quantitatively high-quality matting results. [ABSTRACT FROM PUBLISHER]

Because of real-time requirements and low hardware-cost constraints, conventional TV scalers can only employ basic interpolation technique and thus introduces some artifacts that degrade the viewing quality of the output sequences. In this paper, a low-complexity super-resolution (SR) algorithm, which can provide vivid output image with rich details and sharp edges, and its associated hardware architecture, is proposed. There are two main contributions in this paper. The first is the development of the database-free texture synthesis technique. With the fractal property of nature images, it is possible to find proper high-resolution patches in a low-resolution input image itself. Therefore, the texture synthesis can be performed without database to provide proper and rich details. In addition, a faithful reconstruction constraint is used to maintain the temporal consistency. The second contribution is the hardware architecture design of the database-free texture synthesis. Partial-sum reuse technique is developed to reduce 76% of computation in the texture synthesis, and a tile-based processing technique is proposed to dramatically reduce the on-chip memory and off-chip memory bandwidth requirements. Experimental results show that the proposed SR algorithm outperforms other ones with reasonable hardware cost, where 766k in gate count and 22 kB in on-chip SRAM are required to achieve full high-definition processing ability at the working frequency of 240 MHz. The results show that the proposed algorithm and architecture are able to provide high-quality output in real-time while solving the problems of zigzag and blurred effects caused by the conventional scalers. [ABSTRACT FROM PUBLISHER]

Filtering is widely used in image and video processing for various applications. Recently, the guided filter has been proposed and became one of the popular filtering methods. In this paper, to achieve the computation demand of guided filtering in full-HD video, a double integral image architecture for guided filter ASIC design is proposed. In addition, a reformation of the guided filter formula is proposed, which can prevent the error resulted from truncation in the fractional part and modify the regularization parameter \epsilon on user's demand. The hardware architecture of the guided image filter is then proposed and can be embedded in mobile devices to achieve real-time HD applications. To the best of our knowledge, this paper is also the first ASIC design for guided image filter. With a TSMC 90-nm cell library, the design can operate at 100 MHz and support for Full-HD (1920\,\times\,1080) 30 frame/s with 92.9K gate counts and 3.2 KB on-chip memory. Moreover, for the hardware efficiency, our architecture is also the best compared to other previous works with bilateral filter. [ABSTRACT FROM PUBLISHER]

In this paper, a low-complexity adaptive edge-enhanced algorithm is proposed for the implementation of 2-D image scaling applications. The proposed novel algorithm consists of a linear space-variant edge detector, a low complexity sharpening spatial filter, and a simplified bilinear interpolation. The edge detector is designed to discover the image edges by a low-cost edge-catching technique. The sharpening spatial filter is added as a prefilter to reduce the blurring effect produced by the bilinear interpolation. Furthermore, an adaptive technology is used to enhance the effect of the edge detector by adaptively selecting the input pixels of the bilinear interpolation. In addition, an algebraic manipulation and a hardware sharing techniques are used to simplify bilinear interpolation, which efficiently reduces the computing resources and silicon area in very large scale integration (VLSI) circuits. By adding eight 8-bit registers as a register bank, this design can process streaming data directly and requires only a one-line-buffer memory. The VLSI architecture of this paper contains 6.67-K gate counts and achieves about 280-MHz processing rate by using the TSMC 0.13-um CMOS process. Compared with previous low-complexity techniques, this paper performs with better quality, higher performance, less memory requirements, and lower hardware cost than other image scaling methods. [ABSTRACT FROM PUBLISHER]

Binocular stereo matching is one of the most important algorithms in the field of computer vision. Adaptive support-weight approaches, the current state-of-the-art local methods, produce results comparable to those generated by global methods. However, excessive time consumption is the main problem of these algorithms since the computational complexity is proportionally related to the support window size. In this paper, we present a novel cost aggregation method inspired by domain transformation, a recently proposed dimensionality reduction technique. This transformation enables the aggregation of 2-D cost data to be performed using a sequence of 1-D filters, which lowers computation and memory costs compared to conventional 2-D filters. Experiments show that the proposed method outperforms the state-of-the-art local methods in terms of computational performance, since its computational complexity is independent of the input parameters. Furthermore, according to the experimental results with the Middlebury dataset and real-world images, our algorithm is currently one of the most accurate and efficient local algorithms. [ABSTRACT FROM AUTHOR]

Snakes, or active contours, have been widely used in image processing applications. An external force for snakes called gradient vector flow (GVF) attempts to address traditional snake problems of initialization sensitivity and poor convergence to concavities, while generalized GVF (GGVF) aims to improve GVF snake convergence to long and thin indentations (LTIs). In this paper, we find and show that both GVF and GGVF snakes essentially yield the same performance in capturing LTIs of odd widths, and generally neither can converge to even-width LTIs. Based on a thorough investigation of the GVF and GGVF fields within the LTI during their iterative processes, we identify the crux of the convergence problem, and accordingly propose a novel external force termed as component-normalized GGVF (CN-GGVF) to eliminate the problem. CN-GGVF is obtained by normalizing each component of initial GGVF vectors with respect to its own magnitude. Experimental results and comparisons against GGVF snakes show that the proposed CN-GGVF snakes can capture LTIs regardless of odd or even widths with a remarkably faster convergence speed, while preserving other desirable properties of GGVF snakes with lower computational complexity in vector normalization. [ABSTRACT FROM PUBLISHER]

The H.264/AVC video coding standard can achieve higher compression performance than previous video coding standards, such as MPEG-2, MPEG-4, and H.263. In order to improve the coding performance, the H.264/AVC encoder employs various modes and the rate-distortion (RD) optimization method for selection of an optimum mode. Since the encoder computes RD costs of all possible coding modes to decide the optimum mode, it induces high computational complexity. In this paper, we introduce a fast mode decision method for inter picture macroblocks to reduce the computational complexity. The proposed method significantly reduces the number of candidate modes for the RD optimization process by detecting spatially and temporally homogeneous regions and analyzing motion costs for inter modes and intra prediction costs for intra modes. Experimental results demonstrate that the proposed method drastically reduces the encoding time and the number of the RD cost calculation process while maintaining the high coding efficiency. [ABSTRACT FROM AUTHOR]

This paper proposes a new ringing-artifact reduction method for image resizing in a block discrete cosine transform (DCT) domain. The proposed method reduces ringing artifacts without further blurring, whereas previous approaches must find a compromise between blurring and ringing artifacts. The proposed method consists of DCT-domain filtering and image-domain post-processing, which reduces ripples on smooth regions as well as overshoot near strong edges. By generating a mask map of the overshoot regions, we combine a ripple-reduced image and an overshoot-reduced image according to the mask map in the image domain to obtain a ringing-artifact reduced image. The experimental results show that the proposed method is computationally faster and produces visually finer images than previous ringing-artifact reduction approaches. [ABSTRACT FROM AUTHOR]

Faced with the growing population of seniors, developed countries need to establish new healthcare systems to ensure the safety of elderly people at home. Computer vision provides a promising solution to analyze personal behavior and detect certain unusual events such as falls. In this paper, a new method is proposed to detect falls by analyzing human shape deformation during a video sequence. A shape matching technique is used to track the person's silhouette along the video sequence. The shape deformation is then quantified from these silhouettes based on shape analysis methods. Finally, falls are detected from normal activities using a Gaussian mixture model. This paper has been conducted on a realistic data set of daily activities and simulated falls, and gives very good results (as low as 0% error with a multi-camera setup) compared with other common image processing methods. [ABSTRACT FROM AUTHOR]

This paper presents a novel method for detecting salient regions in both images and videos based on a discriminant center-surround hypothesis that the salient region stands out from its surroundings. To this end, our spatiotemporal approach combines the spatial saliency by computing distances between ordinal signatures of edge and color orientations obtained from the center and the surrounding regions and the temporal saliency by simply computing the sum of absolute difference between temporal gradients of the center and the surrounding regions. Our proposed method is computationally efficient, reliable, and simple to implement and thus it can be easily extended to various applications such as image retargeting and moving object extraction. The proposed method has been extensively tested and the results show that the proposed scheme is effective in detecting saliency compared to various state-of-the-art methods. [ABSTRACT FROM PUBLISHER]

In just noticeable difference (JND) models, evaluation of contrast masking (CM) is a crucial step. More specifically, CM due to edge masking (EM) and texture masking (TM) needs to be distinguished due to the entropy masking property of the human visual system. However, TM is not estimated accurately in the existing JND models since they fail to distinguish TM from EM. In this letter, we propose an enhanced pixel domain JND model with a new algorithm for CM estimation. In our model, total-variation based image decomposition is used to decompose an image into structural image (i.e., cartoon like, piecewise smooth regions with sharp edges) and textural image for estimation of EM and TM, respectively. Compared with the existing models, the proposed one shows its advantages brought by the better EM and TM estimation. It has been also applied to noise shaping and visual distortion gauge, and favorable results are demonstrated by experiments on different images. [ABSTRACT FROM PUBLISHER]

This paper presents an efficient error concealment algorithm for video bitstream over error-prone channel suffering from damage. Moreover, hardware architecture design and chip implementation of the proposed error concealment algorithm are also presented. For spatial error concealment, a mode selection algorithm considering the reuse of intra mode information embedded in bitstream is developed for the adaptation of bilinear and directional interpolation. It suffers only 0.08 dB video quality drop in average but the speedup measured on a general purpose processor is up to 40 times compared with the conventional methods. It is also more suitable for low cost hardware design. For temporal error concealment, the decoded motion vectors of the neighboring blocks of the corrupted macroblock are reused to provide hints to estimate the motion vector of the corrupted macroblock. Moreover, for real-time applications, a data and computational results reuse scheme of motion vector estimation is proposed and 96% computation and memory bandwidth can be reduced compared with the conventional methods with 0.18 dB quality drop in average. With the UMC 90 nm 1P9M process, the proposed error concealment engine can process HDTV1080P 30 frames per second video data and the power consumption is 15.77 mW at 125 MHz operation frequency. [ABSTRACT FROM PUBLISHER]