Search

Another way to demonstrate the accuracy of the component velocity measurements is to show the accuracy with which estimates of 2-d velocity could be computed by a subsequent stage of processing. A linear model for v(x, t) in each local neighbourhood was determined from collections of component velocity estimates in the neighbourhood. As discussed in Chapter 5, this approach is limited because it presupposes that the local component velocities reflect the motion of a single smooth surface in the scene; the model fails whenever there are multiple velocities, or when there are significant outliers, to which least-squares minimization is sensitive. In essence, the approach taken here tacitly assumes that some form of preliminary segmentation of the component velocity estimates has already occurred.

Vision is the process of inferring scene/world properties from images. Our specific concern is with image sequences and spatiotemporal variations in image intensity. It is therefore natural to begin with a discussion of image formation, the ways in which properties of the world are manifested in images as a function of time and space. In other words, what do we expect the time-varying signal to look like? By examining this question we can begin to identify some of the major issues concerning the measurement of image velocity.

This chapter is intended to motivate and introduce a phase-based definition of image velocity, and a corresponding technique for its measurement. To begin, let R(x, t) denote the response of a velocity-tuned band-pass filter. Because the filter kernels, such as Gabor(x,t; k 0, ω0, C) are complex-valued, R(x, t) is also complex-valued and can therefore be expressed as $$ R\left( {x,t} \right) = p\left( {X,t} \right){e^{i\phi \left( {X,t} \right)}},$$ (6.1) where p(x, t) and 0(x, t) denote its amplitude and phase components (1.4): $$ \begin{gathered} p\left( {X,t} \right) = \left| {R\left( {x,t} \right)} \right|, \hfill \\ \phi \left( {X,t} \right) = \arg \left[ {R\left( {X,t} \right)} \right]. \hfill \\ \end{gathered}$$

The central theme of this book was a theoretical and empirical study of recombination and mutation in EAs with the objective of better characterizing the roles of these operators. This theme proceeded in stages. First, static, component-wise analyses of recombination and mutation were performed in isolation. Then, dynamic analyses were performed, which included all aspects of an EA. The results from the static analyses were used to drive the experiments performed in the dynamic analyses. Finally, the results from both the static and dynamic analyses were confirmed empirically with real EAs. This occupied Chaps. 3–8, as well as Chaps. 12 and 14.

This chapter describes the spatiotemporal input signal and models of image velocity using Fourier analysis. It is shown that velocity is a form of orientation in space-time, and has a very simple expression in the frequency domain. We begin with the 2-d translation of textured patterns and 1-d profiles, and then discuss the effects of spatiotemporal localization (windowing), the uncertainty relation, transparency, occlusion, temporal smoothing, sampling and motion blur. This perspective is also important for the design and understanding of linear filters which are characterized by the regions in the frequency domain to which they respond strongly or attenuate. Linear filters can be used as an initial stage of processing to separate image structure according to scale and velocity. The construction of a representation of the input based on a family of velocity-tuned filters is discussed in Chapter 4.

The results of Chapters 9 and 10 concerning the stability of phase information were derived mainly with a white Gaussian noise model of the input. We now consider in more detail their application to real images. The issue of phase stability is discussed in Section 11.1 in the context of natural images that contain salient image features, such as edges, and regions with virtually no structure at all to which the filters respond. We then illustrate the results in the context of binocular disparity, for which matching band-pass filtered versions of the left and right views of a scene along epipolar lines can be viewed as a 1-d problem. After discussing the basic approach in Section 11.2, we consider the accuracy of phase-based disparity measurements for 1-d signals with and without the removal of singularity neighbourhoods, and then as a function of scale and translation perturbations between two views. Section 11.4 briefly illustrates some of these points with 2-d images.

The EiC and Associate EiC express our gratitude to David Forsyth, Brendan Frey, Venu Govindaraju, and Cordelia Schmid who are retiring as associate editors of TPAMI. While we will miss their dedication to the transactions, we hope that they will be enjoying a bit more free time. We are also pleased to announce that Professor Daniel Lopresti, Professor B.S. Manjunath, Professor Marc Pollefeys, and Professor Ramin Zabih have joined the editorial board. Professor Lopresti will oversee papers in document and handwriting analysis, biometrics, approximate string matching algorithms, and performance evaluation. Professor Manjunath will be considering papers in feature extraction, segmentation, image/video retrieval, and image registration. Professor Pollefeys will be responsible for submissions in structure from motion, stereo, multiple view geometry and camera calibration, 3D and appearance modeling, shape-from-X techniques, and novel sensors. Professor Zabih will handle papers on stereo and medical imaging as well as energy minimization and graph algorithms. We look forward to working with them. Their brief biographies appear herein.

Several computational theories of early visual processing, such as Marr's zero-crossing theory, are biologically motivated and based largely on the well-known difference of Gaussians (DOG) receptive-field model of retinal processing. We examine the physiological relevance of the DOG, particularly in the light of evidence indicating significant spatiotemporal inseparability in the behaviour of retinal cell types. From the form of the inseparability we find that commonly accepted functional interpretations of retinal processing based on the DOG, such as the Laplacian of a Gaussian and zero crossings, are not valid for time-varying images. In contrast to current machine-vision approaches, which attempt to separate form and motion information at an early stage, it appears that this is not the case in biological systems. It is further shown that the qualitative form of this inseparability provides a convenient precursor to the extraction of both form and motion information. We show the construction of efficient mechanisms for the extraction of orientation and two-dimensional normal velocity through the use of a hierarchical computational framework. The resultant mechanisms are well localized in space-time and can be easily tuned to various degrees of orientation and speed specificity. Plusieurs theories computationnelles du traitement de la vision primaire, comme la theorie du croisementzero de Marr, sont biologiquement motivees et basees largement sur le celebre modele du champ receptif de la difference de gaussiens (DDG) applique1 au traitement retinien. Nous examinons ici la pertinence physiologique de la DDG, particulierement en tenant compte de ľevidence indiquant une inseparabilite spatio-temporelle significative dans le comportement de la cellule retinienne. A partir des caracteristiques de ľinseparabilite, nous trouvons que les interpretations fonctionnelles classiques du traitement retinien basees sur la DDG, tels que le laplacien ?un gaussien et les croisementszero, ne sont pas valides pour les images variant dans le temps. Contrairement aux approches actuelles en vision automatique, qui tendent a separer dans une premiere etape ľinformation sur la forme de celle sur le mouvement, il apparait que les systemes biologiques fonctionnent tres differemment. On montrera plus loin que la qualitye de cette inseparabilite donne un premier outil approprie pour ľextraction de ľinformation portant a la fois sur la forme et sur le mouvement. Nous decrivons la construction de mecanismes efficients pour ľextraction de ľorientation et de la vitesse 2-d normale a-travers ľutilisation ?un cadre computationnel hierarchique. Les mecanismes resultants sont bien localises dans ľespace-temps et peuvent aisement ětre regies selon divers degres de specification de ľorientation et de la vitesse. Mots cles: vision biologique et automatique, difference de gaussiens, images variant dans le temps, extraction de la vitesse, traitement hierarchique.

We examine the implications of significant inseparable behaviour in centre-surround retinal cell types. From the form of a spatiotemporal centre-surround (CS) model which agrees qualitatively with physiological observations, we find that the sustained/transient dichotomy is a poor distinction for X-type/Y-type retinal ganglion cells since both exhibit inseparability. Static centre-surround models and spatiotemporal separable models are not valid for time-varying stimuli. Our results contradict the models for X- and Y-type ganglion cells proposed by Marr and Hildreth (1980) and Marr and Ullman (1981), and raise doubts about the physiological validity of Marr's zero-crossing theory. The CS filter is an attractive precursor to the extraction of 2-d motion information.

The design of a system that understands visual motion, whether biological or machine, must adhere to certain constraints. These constraints include the types and numbers of available processors, how those processors are arranged, the nature of the task, as well as the characteristics of the input itself. This chapter examines some of these constraints, and in two parts, presents a framework for research in this area. The first part, Section 11.2, involves time complexity arguments demonstrating that the common attack to this problem, namely, an approach that is spatially parallel (as least conceptually), with temporal considerations strictly subsequent to the spatial ones, cannot possibly succeed. The essence of this claim is not a new one, and was motivated by similar comments by Neisser (1967), among others. What Neisser and others did not do, however, is provide a framework that is plausible. Expanding on the time complexity argument, we show that in addition to spatial parallelism, the basic system elements include hierarchical organization through abstraction of both prototypical visual knowledge as well as early representations of the input, and the separation of input measurements into logical feature maps.

This paper concerns the first functional level of visual processing in which low-level primitives are extracted for use in higher levels of processing. We constrain the computational nature of this first level such that a rich description of local intensity structure is computed while requiring no previous or concurrent interpretation. The simultaneous use and interaction of different types of visual information extracted in this way will facilitate a variety higher level tasks. We outline principles for the analysis and design of mechanisms selectively sensitive to local orientation and velocity information. We also discuss tools for the construction of such mechanisms in terms of a hierarchical computational framework. The framework consists of of cascades of explicit (convolution) and implicit (lateral interactions) spatiotemporal processing. The degree of orientation or velocity tuning can be altered by varying the number of layers in the cascade and the form of the processing at each layer.

This paper concerns the first functional level of visual processing in which basic image properties are measured and made available for later interpretation. We constrain this level to be a blind process, that is, an image-independent process which involves no previous or concurrent interpretation. Interpretation is postponed until a rich description of the image, including at least orientation and velocity information, is available. The simultaneous use of such different types of visual information will facilitate most subsequent tasks. For example, in the human visual system, primitive motion and depth information contribute significantly to early form interpretation. The velocity selective mechanisms we propose correspond to short-range motion processing in the human system and require no previous spatial interpretation. By contrast, current approaches to the determination of optic flow in machine vision rely on some degree of spatial interpretation (such as contour estimation, peak finding, or more elaborate spatial analysis). Here, we show the extraction of orientation and 2-D normal velocity based on layers of explicit (bottom-up) and implicit (lateral interactions) spatiotemporal processing. The hierarchy allows a very simple and efficient construction of mechanisms that are well localized in space–time and tuned to narrow ranges of orientation and speed. The degree of specificity can be altered by varying the number of layers in the cascade and the form of processing in each layer.

This paper presents a novel image editing program emphasizing easy selection and manipulation of material found in informal, casual documents such as sketches, handwritten notes, whiteboard images, screen snapshots, and scanned documents. The program, called ScanScribe, offers four significant advances. First, it presents a new, intuitive model for maintaining image objects and groups, along with underlying logic for updating these in the course of an editing session. Second, ScanScribe takes advantage of newly developed image processing algorithms to separate foreground markings from a white or light background, and thus can automatically render the background transparent so that image material can be rearranged without occlusion by background pixels. Third, ScanScribe introduces new interface techniques for selecting image objects with a pointing device without resorting to a palette of tool modes. Fourth, ScanScribe presents a platform for exploiting image analysis and recognition methods to make perceptually significant structure readily available to the user. As a research prototype, ScanScribe has proven useful in the work of members of our laboratory, and has been released on a limited basis for user testing and evaluation.

A framework for learning parameterized models of optical flow from image sequences is presented. A class of motions is represented by a set of orthogonal basis flow fields that are computed from a training set using principal component analysis. Many complex image motions can be represented by a linear combination of a small number of these basis flows. The learned motion models may be used for optical flow estimation and for model-based recognition. For optical flow estimation we describe a robust, multi-resolution scheme for directly computing the parameters of the learned flow models from image derivatives. As examples we consider learning motion discontinuities, non-rigid motion of human mouths, and articulated human motion.

We propose an approach to incorporating dynamic models into the human body tracking process that yields full 3--D reconstructions from monocular sequences. We formulate the tracking problem is terms of minimizing a differentiable criterion whose differential structure is rich enough for successful optimization using a single-hypothesis hill-climbing approach as opposed to a multi-hypotheses probabilistic one. In other words, we avoid the computational complexity of multi-hypotheses algorithms while obtaining excellent results under challenging conditions. To demonstrate this, we focus on monocular tracking of a golf swing from ordinary videos. It involves both dealing with potentially very different swing styles, recovering arm motions that are perpendicular to the camera plane and handling strong self-occlusions.

Phase-based methods for extracting binocular disparity are discussed, including phase-difference methods and phase-correlation. A third method is also described that combines some of their properties, and appears consistent with physiological data. >