Search

Following Marr’s insight, we propose a generative image representation called primal sketch, which integrates two modeling components. The first component explains the structural part of an image, such as ob ject boundaries, by a hidden layer of image primitives. The second component models the remaining textural part without distinguishable elements by Markov random fields that interpolate the structural part of the image. We adopt an artist’s notion by calling the two components “sketchable” and “non-sketchable” parts respectively. A dictionary of image primitives are used for modeling structures in natural images, and each primitive is specified by variables for its photometric, geometric, and topological attributes. The primitives in the image representation are not independent but organized in an sketch graph. This sketch graph is modeled by a spatial Markov model that enforces Gestalt organizations. The inference of the sketch graph consists of two phases. Phase I sequentially adds the most prominent image primitives in a procedure similar to matching pursuit. Phase II edits the sketch graph by a number of graph operators to achieve good Gestalt organizations. Experiments show that the primal sketch model produces satisfactory results for a large number of generic images. The primal sketch model is not only a parsimonious image representation for lossy image coding, but also provides a meaningful mid-level generic representation for other vision tasks.

Vision can be posed as a statistical learning and inference problem. As an over-simplified account, let W be a description of the outside scene in terms of ìwhat is where,î let I be the retina image, and let p(W, I) be the joint distribution of W and I. 1 Then visual learning is to learn p(W, I) from training data, and visual perception is to infer W from I based on p(W|I). There are two major schools on visual learning and perception. One school is operation-oriented and learns the inferential process defined by p(W|I) directly, often in the form of an explicit transformation W ? F(I). This scheme is mostly used in supervised learning, where W is object category, and is given in training data. The other school is representation-oriented and learns the generative process p(W) and p(I|W) explicitly, then perception is to invert the generative process by maximizing or sampling p(W|I) ? p(W)p(I|W). In this scheme, p(W) may also be accounted for by a regularization term such as smoothness or sparsity. This scheme is often used in unsupervised learning where W is not available in training data.

Dynamic textures are sequences of images of moving scenes that exhibit certain stationarity properties in time; these include sea-waves, smoke, foliage, whirlwind etc. We present a novel characterization of dynamic textures that poses the problems of modeling, learning, recognizing and synthesizing dynamic textures on a firm analytical footing. We borrow tools from system identification to capture the “essence” of dynamic textures; we do so by learning (i.e. identifying) models that are optimal in the sense of maximum likelihood or minimum prediction error variance. For the special case of second-order stationary processes, we identify the model sub-optimally in closed-form. Once learned, a model has predictive power and can be used for extrapolating synthetic sequences to infinite length with negligible computa- tional cost. We present experimental evidence that, within our framework, even low-dimensional models can capture very complex visual phenomena.

Recent results on sparse coding and independent component analysis suggest that human vision first represents a visual image by a linear superposition of a relatively small number of localized elongate, oriented image bases. With this representation, the sketch of an image consists of the locations, orientations, and elongations of the images bases, and the sketch can be visually illustrated by depicting each image base by a linelet of the same length and orientation. Built on the insight of sparse and independent component analysis, we propose a two-level generative model for textures. At the bottom-level, the texture image is represented by a linear superposition of image bases. At the top-level, a Markov model is assumed for the placement of the image bases or the sketch, and the model is characterized by a set of simple geometrical feature statistics.

This paper presents a class of statistical models that integrate two statistical modeling paradigms in the literature: I) Descriptive methods, such as Markov random fields and minimax entropy learning [41] and II) Generative methods, such as principal component analysis, independent component analysis [2] transformed component analysis [11], wavelet coding [27, 5], and sparse coding [30, 24]. In this paper, we demonstrate the integrated framwork by constructing a class of hierarchical models for texton patterns ) the term "texton" was coined by psychologist Julez in the early 80's). At the bottom level of the model, we assume that an observed texture image is generated by multiple hidden "texton maps", and textons on each map are translated, scaled, stretched, and oriented versions of a window function, like mini-templates or wavelet bases. The texton maps generate the observed image by occlusion or linear superposition. this bottom level of the model is generative in nature. At the top level of the model, the spatial arrangements of the textons in the texton maps are characterized by minimax entropy principle, which leads to embellished versions of Gibbs point rocess [34]. The top level of the model is descriptive in nature. We demonstrate the integrated model by a set of experiments.

Independent component analysis (ICA) is an important method for blind source separation and unsupervised learning. Recently, the method has been extended to the overcomplete situation where the number of sources is greater than the number of receivers. Comparing complete ICA and overcomplete ICA in existing literature, one can notice that complete ICA does not assume noise in observations, and the sources can be ex- plicitly solved from the receivers, whereas the overcomplete ICA in gen- eral assumes noise in observations and the sources are implicitly solved by gradient type algorithms. In this paper, we present an explicit null- space representation for overcomplete ICA in the noiseless situation based on singular value decomposition (SVD), and develop an algorithm for estimating mixing matrix and recovering the sources. The null-space representation makes the connection between complete ICA and over- complete ICA more apparent, and leads to a mathematical explanation of lateral inhibition in the context of overcomplete linear model. It also appears to work well on the experimental examples. Moreover, it can be extended to the situation where there is noise in observations, and may lead to more efficient algorithms in this situation.

Textons refer to fundamental micro-structures in generic natural images and thus constitute the basic elements in early (pre-attentive) visual perception. However, the word "textong" remains a vague concept in the literature of computer vision and visual perception, and a precist mathematical definition has yet to be found. In this article, we argue that the definition of texton should be governed by a sound mathematical model of images, and the set of textons must be learned from, or best tuned to, an image ensemble. We adopt a generative image model that an image is a superposition of bases from an over-complete dictionary, then a texton is defined as a mini-template that consists of a varying number of image bases with some geometric and photometric configurations. By analogy to physics, if image bases are like protons, neutrons and electrons, then textons are like atoms. Then a small number of textons can be learned from training images as repeating micro-structures. We report four experiments for comparison. The first experiment computes clusters in feature space of filter responses. The second transformed component analysis in both feature space and image patches. The third adopts a two-layer generative model where an image is generated by image bases and image bases are generated by textons. The fourth experiment shows textons from motion image sequences, which we call movetons.

We have learned a lot from studying the sequence of artful works of the two authors on EM/data augmentation. In this note, we will discuss some of our thoughts (or rather speculations) on the problem of vision from the perspective of missing data modeling and data augmentation.