Search

This paper introduces a new model for image texture classification based on wavelet transformation and singular value decomposition. The probability density function of the singular values of wavelet transformation coefficients of image textures is modeled as an exponential function. The model parameter of the exponential function is estimated using maximum likelihood estimation technique. Truncation of lower singular values is employed to classify textures in the presence of noise. Kullback-Leibler distance (KLD) between estimated model parameters of image textures is used as a similarity metric to perform the classification using minimum distance classifier. The exponential function permits us to have closed-form expressions for the estimate of the model parameter and computation of the KLD. These closed-form expressions reduce the computational complexity of the proposed approach. Experimental results are presented to demonstrate the effectiveness of this approach on the entire 111 textures from Brodatz database. The experimental results demonstrate that the proposed approach improves recognition rates using a lower number of parameters on large databases. The proposed approach achieves higher recognition rates compared to the traditional sub-band energy-based approach, the hybrid IMM/SVM approach, and the GGD-based approach.

This paper proposes a novel image-retargeting algorithm that can retarget images to a large family of non-rectangular shapes. Specifically, we study image retargeting from a broader perspective that includes the content as well as the shape of an image, and the proposed content and shape-aware image-retargeting (CASAIR) algorithm is driven by the dual objectives of image content preservation and image domain transformation, with the latter defined by an application-specific target shape. The algorithm is based on the idea of seam segment carving that successively removes low-cost seam segments from the image to simultaneously achieve the two objectives, with the selection of seam segments determined by a cost function incorporating inputs from image content and target shape. To provide a complete characterization of shapes that can be obtained using CASAIR, we introduce the notion of bhv-convex shapes, and we show that bhv-convex shapes are precisely the family of shapes that can be retargeted to by CASAIR. The proposed algorithm is simple in both its design and implementation, and in practice, it offers an efficient and effective retargeting platform that provides its users with considerable flexibility in choosing target shapes. To demonstrate the potential of CASAIR for broadening the application scope of image retargeting, this paper also proposes a smart camera-projector system that incorporates CASAIR. In the context of ubiquitous display, CASAIR equips the camera-projector system with the capability of retargeting images online in order to maximize the quality and fidelity of the displayed images whenever the situation demands.

Over the past three decades, a number of face recognition methods have been proposed in computer vision, and most of them use holistic face images for person identification. In many real-world scenarios especially some unconstrained environments, human faces might be occluded by other objects, and it is difficult to obtain fully holistic face images for recognition. To address this, we propose a new partial face recognition approach to recognize persons of interest from their partial faces. Given a pair of gallery image and probe face patch, we first detect keypoints and extract their local textural features. Then, we propose a robust point set matching method to discriminatively match these two extracted local feature sets, where both the textural information and geometrical information of local features are explicitly used for matching simultaneously. Finally, the similarity of two faces is converted as the distance between these two aligned feature sets. Experimental results on four public face data sets show the effectiveness of the proposed approach.

This paper presents a novel preprocessing method of color-to-gray document image conversion. In contrast to the conventional methods designed for natural images that aim to preserve the contrast between different classes in the converted gray image, the proposed conversion method reduces as much as possible the contrast (i.e., intensity variance) within the text class. It is based on learning a linear filter from a predefined data set of text and background pixels that: 1) when applied to background pixels, minimizes the output response and 2) when applied to text pixels, maximizes the output response, while minimizing the intensity variance within the text class. Our proposed method (called learning-based color-to-gray) is conceived to be used as preprocessing for document image binarization. A data set of 46 historical document images is created and used to evaluate subjectively and objectively the proposed method. The method demonstrates drastically its effectiveness and impact on the performance of state-of-the-art binarization methods. Four other Web-based image data sets are created to evaluate the scalability of the proposed method.

This paper presents a no-reference perceptual metric that quantifies the degree of perceived regularity in textures. The metric is based on the similarity of visual attention (VA) of the textural primitives and the periodic spatial distribution of foveated fixation regions throughout the image. A ground-truth eye-tracking database for textures is also generated as part of this paper and is used to evaluate the performance of the most popular VA models. Using the saliency map generated by the best VA model, the proposed texture regularity metric is computed. It is shown through subjective testing that the proposed metric has a strong correlation with the mean opinion score for the perceived regularity of textures. The proposed texture regularity metric can be used to improve the quality and performance of many image processing applications like texture synthesis, texture compression, and content-based image retrieval.

A new approach to blind image quality assessment (BIQA), requiring no training, is proposed in this paper. The approach is named as blind image quality evaluator based on scales and works by evaluating the global difference of the query image analyzed at different scales with the query image at original resolution. The approach is based on the ability of the natural images to exhibit redundant information over various scales. A distorted image is considered as a deviation from the natural image and bereft of the redundancy present in the original image. The similarity of the original resolution image with its down-scaled version will decrease more when the image is distorted more. Therefore, the dissimilarities of an image with its low-resolution versions are cumulated in the proposed method. We dissolve the query image into its scale-space and measure the global dissimilarity with the co-occurrence histograms of the original and its scaled images. These scaled images are the low pass versions of the original image. The dissimilarity, called low pass error, is calculated by comparing the low pass versions across scales with the original image. The high pass versions of the image in different scales are obtained by Wavelet decomposition and their dissimilarity from the original image is also calculated. This dissimilarity, called high pass error, is computed with the variance and gradient histograms and weighted by the contrast sensitivity function to make it perceptually effective. These two kinds of dissimilarities are combined together to derive the quality score of the query image. This method requires absolutely no training with the distorted image, pristine images, or subjective human scores to predict the perceptual quality but uses the intrinsic global change of the query image across scales. The performance of the proposed method is evaluated across six publicly available databases and found to be competitive with the state-of-the-art techniques.

We propose a new mathematical and algorithmic framework for unsupervised image segmentation, which is a critical step in a wide variety of image processing applications. We have found that most existing segmentation methods are not successful on histopathology images, which prompted us to investigate segmentation of a broader class of images, namely those without clear edges between the regions to be segmented. We model these images as occlusions of random images, which we call textures, and show that local histograms are a useful tool for segmenting them. Based on our theoretical results, we describe a flexible segmentation framework that draws on existing work on nonnegative matrix factorization and image deconvolution. Results on synthetic texture mosaics and real histology images show the promise of the method.

Many computer vision applications, including image classification, matching, and retrieval use global image representations, such as the Fisher vector, to encode a set of local image patches. To describe these patches, many local descriptors have been designed to be robust against lighting changes and noise. However, local image descriptors are unstable when the underlying image signal is low. Such low-signal patches are sensitive to small image perturbations, which might come e.g., from camera noise or lighting effects. In this paper, we first quantify the relation between the signal strength of a patch and the instability of that patch, and second, we extend the standard Fisher vector framework to explicitly take the descriptor instabilities into account. In comparison to common approaches to dealing with descriptor instabilities, our results show that modeling local descriptor instability is beneficial for object matching, image retrieval, and classification.

We introduce an adaptive continuous-domain modeling approach to texture and natural images. The continuous-domain image is assumed to be a smooth function, and we embed it in a parameterized Sobolev space. We point out a link between Sobolev spaces and stochastic auto-regressive models, and exploit it for optimally choosing Sobolev parameters from available pixel values. To this aim, we use exact continuous-to-discrete mapping of the auto-regressive model that is based on symmetric exponential splines. The mapping is computationally efficient, and we exploit it for maximizing an approximated Gaussian likelihood function. We account for non-Gaussian Levy-type processes by deriving a more robust estimator that is based on the sample auto-correlation sequence. Both estimators use multiple initialization values for overcoming the local minima structure of the fitting criteria. Experimental image resizing results indicate that the auto-correlation criterion can cope better with non-Gaussian processes and model mismatch. Our work demonstrates the importance of the auto-correlation function in adaptive image interpolation and image modeling tasks, and we believe it is instrumental in other image processing tasks as well.

The dual-tree quaternion wavelet transform (QWT) is a new multiscale analysis tool for geometric image features. The QWT is a near shift-invariant tight frame representation whose coefficients sport a magnitude and three phases: two phases encode local image shifts while the third contains image texture information. The QWT is based on an alternative theory for the 2-D Hilbert transform and can be computed using a dual-tree filter bank with linear computational complexity. To demonstrate the properties of the QWT's coherent magnitude/phase representation, we develop an efficient and accurate procedure for estimating the local geometrical structure of an image. We also develop a new multiscale algorithm for estimating the disparity between a pair of images that is promising for image registration and flow estimation applications. The algorithm features multiscale phase unwrapping, linear complexity, and sub-pixel estimation accuracy.

A new variational level set approach is proposed for lung nodule segmentation in lung CT scans. A general lung nodule shape model is proposed using implicit spaces as a signed distance function. The shape model is fused with the image intensity statistical information in a variational segmentation framework. The nodule shape model is mapped to the image domain by a global transformation that includes inhomogeneous scales, rotation, and translation parameters. A matching criteria between the shape model and the image implicit representations is employed to handle the alignment process. Transformation parameters evolve through gradient descent optimization to handle the shape alignment process and hence mark the boundaries of the nodule “head.” The embedding process takes into consideration the image intensity as well as prior shape information. A nonparametric density estimation approach is employed to handle the statistical intensity representation of the nodule and background regions. The proposed technique does not depend on nodule type or location. Exhaustive experimental and validation results are demonstrated on 742 nodules obtained from four different CT lung databases, illustrating the robustness of the approach.

This paper proposes a new super-resolution (SR) scheme for landmark images by retrieving correlated web images. Using correlated web images significantly improves the exemplar-based SR. Given a low-resolution (LR) image, we extract local descriptors from its up-sampled version and bundle the descriptors according to their spatial relationship to retrieve correlated high-resolution (HR) images from the web. Though similar in content, the retrieved images are usually taken with different illumination, focal lengths, and shot perspectives, resulting in uncertainty for the HR detail approximation. To solve this problem, we first propose aligning these images to the up-sampled LR image through a global registration, which identifies the corresponding regions in these images and reduces the mismatching. Second, we propose a structure-aware matching criterion and adaptive block sizes to improve the mapping accuracy between LR and HR patches. Finally, these matched HR patches are blended together by solving an energy minimization problem to recover the desired HR image. Experimental results demonstrate that our SR scheme achieves significant improvement compared with four state-of-the-art schemes in terms of both subjective and objective qualities.

Prediction is an essential operation in many image processing applications, such as object detection and image and video compression. When the images are modeled as Gaussian, the optimal predictor is linear and easy to obtain. However, image texture and clutter are often non-Gaussian, and, in such cases, optimal predictors are difficult to obtain. In this paper, we derive an optimal predictor for an important class of non-Gaussian image models, the block-based multivariate Gaussian mixture model. This predictor has a special nonlinear structure: it is a linear combination of the neighboring pixels, but the combination coefficients are also functions of the neighboring pixels, not constants. The efficacy of this predictor is demonstrated in object detection experiments where the prediction error image is used to identify "hidden" objects. Experimental results indicate that when the background texture is nonlinear, i.e., with fast-switching gray-level patches, it performs significantly better than the optimal linear predictor.

Segmentation of the target object(s) from images that have multiple complicated regions, mixture intensity distributions or are corrupted by noise poses a challenge for the level set models. In addition, the conventional piecewise smooth level set models normally require prior knowledge about the number of image segments. To address these problems, we propose a novel segmentation energy function with two distribution descriptors to model the background and the target. The single background descriptor models the heterogeneous background with multiple regions. Then, the target descriptor takes into account the intensity distribution and incorporates local spatial constraint. Our descriptors, which have more complete distribution information, construct the unique energy function to differentiate the target from the background and are more tolerant of image noise. We compare our approach to three other level set models: 1) the Chan-Vese; 2) the multiphase level set; and 3) the geodesic level set. This comparison using 260 synthetic images with varying levels and types of image noise and medical images with more complicated backgrounds showed that our method outperforms these models for accuracy and immunity to noise. On an additional set of 300 synthetic images, our model is also less sensitive to the contour initialization as well as to different types and levels of noise.

Despite a significant growth in the last few years, the availability of 3D content is still dwarfed by that of its 2D counterpart. To close this gap, many 2D-to-3D image and video conversion methods have been proposed. Methods involving human operators have been most successful but also time-consuming and costly. Automatic methods, which typically make use of a deterministic 3D scene model, have not yet achieved the same level of quality for they rely on assumptions that are often violated in practice. In this paper, we propose a new class of methods that are based on the radically different approach of learning the 2D-to-3D conversion from examples. We develop two types of methods. The first is based on learning a point mapping from local image/video attributes, such as color, spatial position, and, in the case of video, motion at each pixel, to scene-depth at that pixel using a regression type idea. The second method is based on globally estimating the entire depth map of a query image directly from a repository of 3D images ( image+depth pairs or stereopairs) using a nearest-neighbor regression type idea. We demonstrate both the efficacy and the computational efficiency of our methods on numerous 2D images and discuss their drawbacks and benefits. Although far from perfect, our results demonstrate that repositories of 3D content can be used for effective 2D-to-3D image conversion. An extension to video is immediate by enforcing temporal continuity of computed depth maps.

Histogram transformation defines a class of image processing operations that are widely applied in the implementation of data normalization algorithms. In this paper, we present a new variational approach for image enhancement that is constructed to alleviate the intensity saturation effects that are introduced by standard contrast enhancement (CE) methods based on histogram equalization. In this paper, we initially apply total variation (TV) minimization with a L1 fidelity term to decompose the input image with respect to cartoon and texture components. Contrary to previous papers that rely solely on the information encompassed in the distribution of the intensity information, in this paper, the texture information is also employed to emphasize the contribution of the local textural features in the CE process. This is achieved by implementing a nonlinear histogram warping CE strategy that is able to maximize the information content in the transformed image. Our experimental study addresses the CE of a wide variety of image data and comparative evaluations are provided to illustrate that our method produces better results than conventional CE strategies.

A fast and effective image fusion method is proposed for creating a highly informative fused image through merging multiple images. The proposed method is based on a two-scale decomposition of an image into a base layer containing large scale variations in intensity, and a detail layer capturing small scale details. A novel guided filtering-based weighted average technique is proposed to make full use of spatial consistency for fusion of the base and detail layers. Experimental results demonstrate that the proposed method can obtain state-of-the-art performance for fusion of multispectral, multifocus, multimodal, and multiexposure images.

Segmentation of text from badly degraded document images is a very challenging task due to the high inter/intra-variation between the document background and the foreground text of different document images. In this paper, we propose a novel document image binarization technique that addresses these issues by using adaptive image contrast. The adaptive image contrast is a combination of the local image contrast and the local image gradient that is tolerant to text and background variation caused by different types of document degradations. In the proposed technique, an adaptive contrast map is first constructed for an input degraded document image. The contrast map is then binarized and combined with Canny's edge map to identify the text stroke edge pixels. The document text is further segmented by a local threshold that is estimated based on the intensities of detected text stroke edge pixels within a local window. The proposed method is simple, robust, and involves minimum parameter tuning. It has been tested on three public datasets that are used in the recent document image binarization contest (DIBCO) 2009 & 2011 and handwritten-DIBCO 2010 and achieves accuracies of 93.5%, 87.8%, and 92.03%, respectively, that are significantly higher than or close to that of the best-performing methods reported in the three contests. Experiments on the Bickley diary dataset that consists of several challenging bad quality document images also show the superior performance of our proposed method, compared with other techniques.

The problem of blind noise level estimation arises in many image processing applications, such as denoising, compression, and segmentation. In this paper, we propose a new noise level estimation method on the basis of principal component analysis of image blocks. We show that the noise variance can be estimated as the smallest eigenvalue of the image block covariance matrix. Compared with 13 existing methods, the proposed approach shows a good compromise between speed and accuracy. It is at least 15 times faster than methods with similar accuracy, and it is at least two times more accurate than other methods. Our method does not assume the existence of homogeneous areas in the input image and, hence, can successfully process images containing only textures.

Printers employing electrophotographic technology typically use clustered-dot screening to avoid potential artifacts caused due to unstable dot rendering. Periodic clustered-dot screens are quite smooth, but also suffer from periodic moire artifacts due to interference with other color channels. Stochastic, clustered-dot screens provide an alternative solution. In this paper, we introduce a new approach for stochastic, clustered-dot halftoning and screen design based on direct binary search. The method deviates from conventional DBS in its use of different filters in different phases of the algorithm. We provide an intuitive explanation for the clustering achieved as a result of this modification, and also discuss ways to control the coarseness of clusters. Due to the non-applicability of conventional screen design procedures to the new halftoning method, we propose a new method for screen design also. Results include images of individually designed constant-tone halftones and halftones of screened folded ramps.

In this paper, an unsupervised model-based texture reproduction technique is described. In accordance with the Julesz's (1962) conjecture, the statistical properties of the prototype up to the second order are copied in order to generate a synthetic texture perceptually indistinguishable from the given sample. However, this task is accomplished using a hybrid approach which operates partially in the spatial domain and partially in a multiresolution domain. The latter employed is the circular harmonic function (CHF) domain since it has been proven to be well suited for mimicking the behavior of the human visual system (HVS). This approach allows, for a wide range of textures typologies, obtaining synthetic textures that better match the prototype with respect to the ones obtained using techniques based on the Julesz's conjecture operating only in the spatial domain, and to dramatically reduce the computational complexity of similar methods operating only in the multiresolution domain.

We introduce a new image texture segmentation algorithm, HMTseg, based on wavelets and the hidden Markov tree (HMT) model. The HMT is a tree-structured probabilistic graph that captures the statistical properties of the coefficients of the wavelet transform. Since the HMT is particularly well suited to images containing singularities (edges and ridges), it provides a good classifier for distinguishing between textures. Utilizing the inherent tree structure of the wavelet HMT and its fast training and likelihood computation algorithms, we perform texture classification at a range of different scales. We then fuse these multiscale classifications using a Bayesian probabilistic graph to obtain reliable final segmentations. Since HMTseg works on the wavelet transform of the image, it can directly segment wavelet-compressed images without the need for decompression into the space domain. We demonstrate the performance of HMTseg with synthetic, aerial photo, and document image segmentations.

In this paper, we propose a simple but effective shadow removal method using a single input image. We first derive a 2-D intrinsic image from a single RGB camera image based solely on colors, particularly chromaticity. We next present a method to recover a 3-D intrinsic image based on bilateral filtering and the 2-D intrinsic image. The luminance contrast in regions with similar surface reflectance due to geometry and illumination variances is effectively reduced in the derived 3-D intrinsic image, while the contrast in regions with different surface reflectance is preserved. However, the intrinsic image contains incorrect luminance values. To obtain the correct luminance, we decompose the input RGB image and the intrinsic image. Each image is decomposed into a base layer and a detail layer. We obtain a shadow-free image by combining the base layer from the input RGB image and the detail layer from the intrinsic image such that the details of the intrinsic image are transferred to the input RGB image from which the correct luminance values can be obtained. Unlike previous methods, the presented technique is fully automatic and does not require shadow detection.

Data-level fusion is believed to have the potential for enhancing human face recognition. However, due to a number of challenges, current techniques have failed to achieve its full potential. We propose spatially optimized data/pixel-level fusion of 3-D shape and texture for face recognition. Fusion functions are objectively optimized to model expression and illumination variations in linear subspaces for invariant face recognition. Parameters of adjacent functions are constrained to smoothly vary for effective numerical regularization. In addition to spatial optimization, multiple nonlinear fusion models are combined to enhance their learning capabilities. Experiments on the FRGC v2 data set show that spatial optimization, higher order fusion functions, and the combination of multiple such functions systematically improve performance, which is, for the first time, higher than score-level fusion in a similar experimental setup.

Intensity inhomogeneity often occurs in real-world images, which presents a considerable challenge in image segmentation. The most widely used image segmentation algorithms are region-based and typically rely on the homogeneity of the image intensities in the regions of interest, which often fail to provide accurate segmentation results due to the intensity inhomogeneity. This paper proposes a novel region-based method for image segmentation, which is able to deal with intensity inhomogeneities in the segmentation. First, based on the model of images with intensity inhomogeneities, we derive a local intensity clustering property of the image intensities, and define a local clustering criterion function for the image intensities in a neighborhood of each point. This local clustering criterion function is then integrated with respect to the neighborhood center to give a global criterion of image segmentation. In a level set formulation, this criterion defines an energy in terms of the level set functions that represent a partition of the image domain and a bias field that accounts for the intensity inhomogeneity of the image. Therefore, by minimizing this energy, our method is able to simultaneously segment the image and estimate the bias field, and the estimated bias field can be used for intensity inhomogeneity correction (or bias correction). Our method has been validated on synthetic images and real images of various modalities, with desirable performance in the presence of intensity inhomogeneities. Experiments show that our method is more robust to initialization, faster and more accurate than the well-known piecewise smooth model. As an application, our method has been used for segmentation and bias correction of magnetic resonance (MR) images with promising results.

Human observers understand the content of an image intuitively. Based upon image content, they perform many image-related tasks, such as creating slide shows and photo albums, and organizing their image archives. For example, to select photos for an album, people assess image quality based upon the main objects in the image. They modify colors in an image based upon the color of important objects, such as sky, grass or skin. Serious photographers might modify each object separately. Photo applications, in contrast, use low-level descriptors to guide similar tasks. Typical descriptors, such as color histograms, noise level, JPEG artifacts and overall sharpness, can guide an imaging application and safeguard against blunders. However, there is a gap between the outcome of such operations and the same task performed by a person. We believe that the gap can be bridged by automatically understanding the content of the image. This paper presents algorithms for automatic tagging of perceptual objects in images, including sky, skin, and foliage, which constitutes an important step toward this goal.

Finding meaningful groupings of image primitives has been a long-standing problem in computer vision. This paper studies how salient groupings can be produced using established theories in the field of visual perception alone. The major contribution is a novel definition of the Gestalt principle of Prägnanz, based upon Koffka's definition that image descriptions should be both stable and simple. Our method is global in the sense that it operates over all primitives in an image at once. It works regardless of the type of image primitives and is generally independent of image properties such as intensity, color, and texture. A novel experiment is designed to quantitatively evaluate the groupings outputs by our method, which takes human disagreement into account and is generic to outputs of any grouper. We also demonstrate the value of our method in an image segmentation application and quantitatively show that segmentations deliver promising results when benchmarked using the Berkeley Segmentation Dataset (BSDS).

Image segmentation plays an important role in computer vision and image analysis. In this paper, image segmentation is formulated as a labeling problem under a probability maximization framework. To estimate the label configuration, an iterative optimization scheme is proposed to alternately carry out the maximum a posteriori (MAP) estimation and the maximum likelihood (ML) estimation. The MAP estimation problem is modeled with Markov random fields (MRFs) and a graph cut algorithm is used to find the solution to the MAP estimation. The ML estimation is achieved by computing the means of region features in a Gaussian model. Our algorithm can automatically segment an image into regions with relevant textures or colors without the need to know the number of regions in advance. Its results match image edges very well and are consistent with human perception. Comparing to six state-of-the-art algorithms, extensive experiments have shown that our algorithm performs the best.

Colorant-based direct binary search (CB-DBS) halftoning proposed in provides an image quality benchmark for dispersed-dot halftoning algorithms. The objective of this paper is to further push the image quality limit. An algorithm called hierarchical colorant-based direct binary search (HCB-DBS) is developed in this paper. By appropriately integrating yellow colorant into dot-overlapping and dot-positioning controls, it is demonstrated that HCB-DBS can achieve better halftone texture of both individual and joint dot-color planes, without compromising the dot distribution of more visible halftone of cyan and magenta colorants. The input color specification is first converted from colorant space to dot-color space with minimum brightness variation principle for full dot-overlapping control. The dot-colors are then split into groups based upon dot visibility. Hierarchical monochrome DBS halftoning is applied to make dot-positioning decision for each group, constrained on the already generated halftone of the groups with higher priority. And dot-coloring is decided recursively with joint monochrome DBS halftoning constrained on the related total dot distribution. Experiments show HCB-DBS improves halftone texture for both individual and joint dot-color planes. And it reduces the halftone graininess and free of color mottle artifacts, comparing to CB-DBS.

A texture representation should corroborate various functions of a texture. In this paper, we present a novel approach that incorporates texture features for retrieval in an examplar-based texture compaction and synthesis algorithm. The original texture is compacted and compressed in the encoder to obtain a thumbnail texture, which the decoder then synthesizes to obtain a perceptually high quality texture. We propose using a probabilistic framework based on the generalized EM algorithm to analyze the solutions of the approach. Our experiment results show that a high quality synthesized texture can be generated in the decoder from a compressed thumbnail texture. The number of bits in the compressed thumbnail is 400 times lower than that in the original texture and 50 times lower than that needed to compress the original texture using JPEG2000. We also show that, in terms of retrieval and synthesization, our compressed and compacted textures perform better than compressed cropped textures and compressed compacted textures derived by the patchwork algorithm.

This study presents a rapid image completion system comprising a training (or analysis) process and an image completion (or synthesis) process. The proposed system adopts a multiresolution approach, which not only improves the convergence rate of the synthesis process, but also provides the ability to deal with large replaced regions. In the training process, a down-sampling approach is applied to create a patch-based texture eigenspace based on multiresolution background region information. In the image completion process, an up-sampling approach is applied to synthesize the replaced foreground regions. To ensure the continuity of the geometric texture structure between the original background scene regions and the replaced foreground regions, directional and nondirectional image completion approaches are developed to reconstruct the global geometric structure and to enhance the local detailed features of the replaced foreground regions in the lower and higher resolution level images, respectively. Moreover, the synthesis priority order of the individual patches and the appropriate choice of completion scheme (i.e., directional or nondirectional) are both determined in accordance with a Hessian matrix decision value (HMDV) parameter. Finally, a texture refinement process is performed to optimize the resolution of the synthesized result.

In this paper, we propose an interactive color natural image segmentation method. The method integrates color feature with multiscale nonlinear structure tensor texture (MSNST) feature and then uses GrabCut method to obtain the segmentations. The MSNST feature is used to describe the texture feature of an image and integrated into GrabCut framework to overcome the problem of the scale difference of textured images. In addition, we extend the Gaussian Mixture Model (GMM) to MSNST feature and GMM based on MSNST is constructed to describe the energy function so that the texture feature can be suitably integrated into GrabCut framework and fused with the color feature to achieve the more superior image segmentation performance than the original GrabCut method. For easier implementation and more efficient computation, the symmetric KL divergence is chosen to produce the estimates of the tensor statistics instead of the Riemannian structure of the space of tensor. The Conjugate norm was employed using Locality Preserving Projections (LPP) technique as the distance measure in the color space for more discriminating power. An adaptive fusing strategy is presented to effectively adjust the mixing factor so that the color and MSNST texture features are efficiently integrated to achieve more robust segmentation performance. Last, an iteration convergence criterion is proposed to reduce the time of the iteration of GrabCut algorithm dramatically with satisfied segmentation accuracy. Experiments using synthesis texture images and real natural scene images demonstrate the superior performance of our proposed method.

Estimation of local orientations in multivariate signals is an important problem in image processing and computer vision. This general problem formulation also covers optical flow estimation, which can be regarded as orientation estimation in space-time-volumes. Modelling a signal using only a single orientation, however, is often too restrictive, since occlusions and transparencies occur frequently, thus necessitating the modelling and analysis of multiple orientations. We, therefore, develop a unifying mathematical model for multiple orientations: Beyond describing an arbitrary number of orientations in scalar- and vector-valued image data such as color image sequences, it allows the unified treatment of additively and occludingly superimposed oriented structures as well as of combinations of these. Based on this model, we describe estimation schemes for an arbitrary number of additively or occludingly superimposed orientations in images. We confirm the performance of our framework on both synthetic and real image data.

We propose a new approach to quantitatively analyze the rendering quality of image-based rendering (IBR) algorithms with depth information. The resulting error bounds for synthesized views depend on IBR configurations including the depth and intensity estimate errors, the scene geometry and texture, the number of actual cameras, their positions and resolution. Specifically, the IBR error is bounded by the summation of three terms, highlighting the impact of using multiple actual cameras, the impact of the noise level at the actual cameras, and the impact of the depth accuracy. We also quantify the impact of occlusions and intensity discontinuities. The proposed methodology is applicable to a large class of common IBR algorithms and can be applied locally. Experiments with synthetic and real scenes show that the developed error bounds accurately characterize the rendering errors. In particular, the error bounds correctly characterize the decay rates of synthesized views' mean absolute errors as O(lambda(-1)) and O(lambda(-2)), where lambda is the local density of actual samples, for 2-D and 3-D scenes, respectively. Finally, we discuss the implications of the proposed analysis on camera placement, budget allocation, and bit allocation.

A novel preferential image segmentation method is proposed that performs image segmentation and object recognition using mathematical morphologies. The method preferentially segments objects that have intensities and boundaries similar to those of objects in a database of prior images. A tree of shapes is utilized to represent the content distributions in images, and curve matching is applied to compare the boundaries. The algorithm is invariant to contrast change and similarity transformations of translation, rotation and scale. A performance evaluation of the proposed method using a large image dataset is provided. Experimental results show that the proposed approach is promising for applications such as object segmentation and video tracking with cluttered backgrounds.

Glass patterns have been exhaustively studied both in the vision literature and from a purely mathematical point of view. We extend the related formalism to the continuous case and we show that continuous Glass patterns can be used for artistic imaging applications. The general idea is to replace natural texture present in an input image with synthetic painterly texture that is generated by means of a continuous Glass pattern, whose geometrical structure is controlled by the gradient orientation of the input image. The behavior of the proposed algorithm is analytically interpreted in terms of the theory of dynamical systems. Experimental results on a broad range of input images validate the effectiveness of the proposed method in terms of lack of undesired artifacts, which are present with other existing methods, and easy interpretability of the input parameters.

This paper proposes a face hallucination method for the reconstruction of high-resolution facial images from single-frame, low-resolution facial images. The proposed method has been derived from example-based hallucination methods and morphable face models. First, we propose a recursive error back-projection method to compensate for residual errors, and a region-based reconstruction method to preserve characteristics of local facial regions. Then, we define an extended morphable face model, in which an extended face is composed of the interpolated high-resolution face from a given low-resolution face, and its original high-resolution equivalent. Then, the extended face is separated into an extended shape and an extended texture. We performed various hallucination experiments using the MPI, XM2VTS, and KF databases, compared the reconstruction errors, structural similarity index, and recognition rates, and showed the effects of face detection errors and shape estimation errors. The encouraging results demonstrate that the proposed methods can improve the performance of face recognition systems. Especially the proposed method can enhance the resolution of single-frame, low-resolution facial images.

In this paper, we present the adaptive bilateral filter (ABF) for sharpness enhancement and noise removal. The ABF sharpens an image by increasing the slope of the edges without producing overshoot or undershoot. It is an approach to sharpness enhancement that is fundamentally different from the unsharp mask (USM). This new approach to slope restoration also differs significantly from previous slope restoration algorithms in that the ABF does not involve detection of edges or their orientation, or extraction of edge profiles. In the ABF, the edge slope is enhanced by transforming the histogram via a range filter with adaptive offset and width. The ABF is able to smooth the noise, while enhancing edges and textures in the image. The parameters of the ABF are optimized with a training procedure. ABF restored images are significantly sharper than those restored by the bilateral filter. Compared with an USM based sharpening method-the optimal unsharp mask (OUM), ABF restored edges are as sharp as those rendered by the OUM, but without the halo artifacts that appear in the OUM restored image. In terms of noise removal, ABF also outperforms the bilateral filter and the OUM. We demonstrate that ABF works well for both natural images and text images.

Two important visual properties of paintings and painting-like images are the absence of texture details and the increased sharpness of edges as compared to photographic images. Painting-like artistic effects can be achieved from photographic images by filters that smooth out texture details, while preserving or enhancing edges and corners. However, not all edge preserving smoothers are suitable for this purpose. We present a simple nonlinear local operator that generalizes both the well known Kuwahara filter and the more general class of filters known in the literature as ldquocriterion and value filter structure.rdquo This class of operators suffers from intrinsic theoretical limitations which give rise to a dramatic instability in presence of noise, especially on shadowed areas. Such limitations are discussed in the paper and overcome by the proposed operator. A large variety of experimental results shows that the output of the proposed operator is visually similar to a painting. Comparisons with existing techniques on a large set of natural images highlight conditions on which traditional edge preserving smoothers fail, whereas our approach produces good results. In particular, unlike many other well established approaches, the proposed operator is robust to degradations of the input image such as blurring and noise contamination.

A framework for inpainting missing parts of a video sequence recorded with a moving or stationary camera is presented in this work. The region to be inpainted is general: it may be still or moving, in the background or in the foreground, it may occlude one object and be occluded by some other object. The algorithm consists of a simple preprocessing stage and two steps of video inpainting. In the preprocessing stage, we roughly segment each frame into foreground and background. We use this segmentation to build three image mosaics that help to produce time consistent results and also improve the performance of the algorithm by reducing the search space. In the first video inpainting step, we reconstruct moving objects in the foreground that are "occluded" by the region to be inpainted. To this end, we fill the gap as much as possible by copying information from the moving foreground in other frames, using a priority-based scheme. In the second step, we inpaint the remaining hole with the background. To accomplish this, we first align the frames and directly copy when possible. The remaining pixels are filled in by extending spatial texture synthesis techniques to the spatiotemporal domain. The proposed framework has several advantages over state-of-the-art algorithms that deal with similar types of data and constraints. It permits some camera motion, is simple to implement, fast, does not require statistical models of background nor foreground, works well in the presence of rich and cluttered backgrounds, and the results show that there is no visible blurring or motion artifacts. A number of real examples taken with a consumer hand-held camera are shown supporting these findings.

We present a method for segmenting images consisting of texture and nontexture regions based on local spectral histograms. Defined as a vector consisting of marginal distributions of chosen filter responses, local spectral histograms provide a feature statistic for both types of regions. Using local spectral histograms of homogeneous regions, we decompose the segmentation process into three stages. The first is the initial classification stage, where probability models for homogeneous texture and nontexture regions are derived and an initial segmentation result is obtained by classifying local windows. In the second stage, we give an algorithm that iteratively updates the segmentation using the derived probability models. The third is the boundary localization stage, where region boundaries are localized by building refined probability models that are sensitive to spatial patterns in segmented regions. We present segmentation results on texture as well as nontexture images. Our comparison with other methods shows that the proposed method produces more accurate segmentation results.

Denoising algorithms based on gradient dependent regularizers, such as nonlinear diffusion processes and total variation denoising, modify images towards piecewise constant functions. Although edge sharpness and location is well preserved, important information, encoded in image features like textures or certain details, is often compromised in the process of denoising. We propose a mechanism that better preserves fine scale features in such denoising processes. A basic pyramidal structure-texture decomposition of images is presented and analyzed. A first level of this pyramid is used to isolate the noise and the relevant texture components in order to compute spatially varying constraints based on local variance measures. A variational formulation with a spatially varying fidelity term controls the extent of denoising over image regions. Our results show visual improvement as well as an increase in the signal-to-noise ratio over scalar fidelity term processes. This type of processing can be used for a variety of tasks in partial differential equation-based image processing and computer vision, and is stable and meaningful from a mathematical viewpoint.

We propose the concept of quality-aware image , in which certain extracted features of the original (high-quality) image are embedded into the image data as invisible hidden messages. When a distorted version of such an image is received, users can decode the hidden messages and use them to provide an objective measure of the quality of the distorted image. To demonstrate the idea, we build a practical quality-aware image encoding, decoding and quality analysis system, which employs: 1) a novel reduced-reference image quality assessment algorithm based on a statistical model of natural images and 2) a previously developed quantization watermarking-based data hiding technique in the wavelet transform domain.

Today's multimedia applications demand sophisticated compression and classification techniques in order to store, transmit, and retrieve audio-visual information efficiently. Over the last decade, perceptually based image compression methods have been gaining importance. These methods take into account the abilities (and the limitations) of human visual perception (HVP) when performing compression. The upcoming MPEG 7 standard also addresses the need for succinct classification and indexing of visual content for efficient retrieval. However, there has been no research that has attempted to exploit the characteristics of the human visual system to perform both compression and classification jointly. One area of HVP that has unexplored potential for joint compression and classification is spatial frequency perception. Spatial frequency content that is perceived by humans can be characterized in terms of three parameters, which are: 1) magnitude; 2) phase; and 3) orientation. While the magnitude of spatial frequency content has been exploited in several existing image compression techniques, the novel contribution of this paper is its focus on the use of phase coherence for joint compression and classification in the wavelet domain. Specifically, this paper describes a human visual system-based method for measuring the degree to which an image contains coherent (perceptible) phase information, and then exploits that information to provide joint compression and classification. Simulation results that demonstrate the efficiency of this method are presented.

In this contribution, image processing techniques are investigated to robustly segment faint striation patterns from mainly isotropic background areas. Whereas well-known procedures to determine local anisotropy can be applied directly to single images, this paper describes an alternative strategy that uses an illumination series. The series is obtained with a spot illumination whose azimuth is varied systematically. The technique is based on the characteristics of the local contrast with respect to the azimuth of the illumination: Whereas pronounced maxima of the local contrast can be observed when the striae are illuminated perpendicularly, an isotropic background texture shows less distinct maxima for random azimuth angles. The qualification of the approach is demonstrated on the segmentation of faint tool marks in forensic science. Experimental results show that the methodology ensures the correct segmentation of such marks.

Multifractal analysis is becoming more and more popular in image segmentation community, in which the box-counting based multifractal dimension estimations are most commonly used. However, in spite of its computational efficiency, the regular partition scheme used by various box-counting methods intrinsically produces less accurate results. In this paper, a novel multifractal estimation algorithm based on mathematical morphology is proposed and a set of new multifractal descriptors, namely the local morphological multifractal exponents is defined to characterize the local scaling properties of textures. A series of cubic structure elements and an iterative dilation scheme are utilized so that the computational complexity of the morphological operations can be tremendously reduced. Both the proposed algorithm and the box-counting based methods have been applied to the segmentation of texture mosaics and real images. The comparison results demonstrate that the morphological multifractal estimation can differentiate texture images more effectively and provide more robust segmentations.

Measurement of visual quality is of fundamental importance to numerous image and video processing applications. The goal of quality assessment (QA) research is to design algorithms that can automatically assess the quality of images or videos in a perceptually consistent manner. Image QA algorithms generally interpret image quality as fidelity or similarity with a "reference" or "perfect" image in some perceptual space. Such "full-reference" QA methods attempt to achieve consistency in quality prediction by modeling salient physiological and psychovisual features of the human visual system (HVS), or by signal fidelity measures. In this paper, we approach the image QA problem as an information fidelity problem. Specifically, we propose to quantify the loss of image information to the distortion process and explore the relationship between image information and visual quality. QA systems are invariably involved with judging the visual quality of "natural" images and videos that are meant for "human consumption." Researchers have developed sophisticated models to capture the statistics of such natural signals. Using these models, we previously presented an information fidelity criterion for image QA that related image quality with the amount of information shared between a reference and a distorted image. In this paper, we propose an image information measure that quantifies the information that is present in the reference image and how much of this reference information can be extracted from the distorted image. Combining these two quantities, we propose a visual information fidelity measure for image QA. We validate the performance of our algorithm with an extensive subjective study involving 779 images and show that our method outperforms recent state-of-the-art image QA algorithms by a sizeable margin in our simulations. The code and the data from the subjective study are available at the LIVE website.

This paper addresses two challenging issues in unsupervised multiscale texture segmentation: determining adequate spatial and feature resolutions for different regions of the image, and utilizing information across different scales/resolutions. The center of a homogeneous texture is analyzed using coarse spatial resolution, and its border is detected using fine spatial resolution so as to locate the boundary accurately. The extraction of texture features is achieved via a multiresolution pyramid. The feature values are integrated across scales/resolutions adaptively. The number of textures is determined automatically using the variance ratio criterion. Experimental results on synthetic and real images demonstrate the improvement in performance of the proposed multiscale scheme over single scale approaches.

Self-similar random fields are of interest in various areas of image processing since they fit certain types of natural patterns and textures. Current treatments of self-similarity in continuous two-dimensional (2-D) space use a definition that is a direct extension of the one-dimensional definition, which requires invariance of the statistics of a random process to time scaling. Current discrete-space 2-D approaches do not consider scaling, but, instead, are based on ad hoc formulations, such as digitizing continuous random fields. In this paper, we show that the current statistical self-similarity definition in continuous space is restrictive and provide an alternative, more general definition. We also provide a formalism for discrete-space statistical self-similarity that relies on a new scaling operator for discrete images. Within the new framework, it is possible to synthesize a wider class of discrete-space self-similar random fields and texture images.