Search

In this paper, we propose a large-scale video based animal counting dataset collected by drones (AnimalDrone) for agriculture and wildlife protection. The dataset consists of two subsets, i.e. , PartA captured on site by drones and PartB collected from the Internet, with rich annotations of more than 4 million objects in 53, 644 frames and corresponding attributes in terms of density, altitude and view. Moreover, we develop a new graph regularized flow attention network (GFAN) to perform density map estimation in dense crowds of video clips with arbitrary crowd density, perspective, and flight altitude. Specifically, our GFAN method leverages optical flow to warp the multi-scale feature maps in sequential frames to exploit the temporal relations, and then combines the enhanced features to predict the density maps. Moreover, we introduce the multi-granularity loss function including pixel-wise density loss and region-wise count loss to enforce the network to concentrate on discriminative features for different scales of objects. Meanwhile, the graph regularizer is imposed on the density maps of multiple consecutive frames to maintain temporal coherency. Extensive experiments are conducted to demonstrate the effectiveness of the proposed method, compared with several state-of-the-art counting algorithms. The AnimalDrone dataset is available at https://github.com/VisDrone/AnimalDrone .

Fluid motion estimation from time-sequenced images is a significant image analysis task. Its application is widespread in experimental fluidics research and many related areas like biomedical engineering and atmospheric sciences. In this paper, we present a novel flow computation framework to estimate the flow velocity vectors from two consecutive image frames. In an energy minimization-based flow computation, we propose a novel data fidelity term, which: 1) can accommodate various measures, such as cross-correlation or sum of absolute or squared differences of pixel intensities between image patches; 2) has a global mechanism to control the adverse effect of outliers arising out of motion discontinuities, proximity of image borders; and 3) can go hand-in-hand with various spatial smoothness terms. Further, the proposed data term and related regularization schemes are both applicable to dense and sparse flow vector estimations. We validate these claims by numerical experiments on benchmark flow data sets.

In this paper we propose to jointly segment and register objects of interest in layered images. Layered imaging refers to imageries taken from different perspectives and possibly by different sensors. Registration and segmentation are therefore the two main tasks which contribute to the bottom level, data alignment, of the multisensor data fusion hierarchical structures. Most exploitations of two layered images assumed that scanners are at very high altitudes and that only one transformation ties the two images. Our data are however taken at mid-range and therefore requires segmentation to assist us examining different object regions in a divide-and-conquer fashion. Our approach is a combination of multiphase active contour method with a joint segmentation-registration technique (which we called MPJSR) carried out in a local moving window prior to a global optimization. To further address layered video sequences and tracking objects in frames, we propose a simple adaptation of optical flow calculations along the active contours in a pair of layered image sequences. The experimental results show that the whole integrated algorithm is able to delineate the objects of interest, align them for a pair of layered frames and keep track of the objects over time.

Face recognition is one of the most intensively studied topics in computer vision and pattern recognition, but few are focused on how to robustly recognize faces with expressions under the restriction of one single training sample per class. A constrained optical flow algorithm, which combines the advantages of the unambiguous correspondence of feature point labeling and the flexible representation of optical flow computation, has been developed for face recognition from expressional face images. In this paper, we propose an integrated face recognition system that is robust against facial expressions by combining information from the computed intraperson optical flow and the synthesized face image in a probabilistic framework. Our experimental results show that the proposed system improves the accuracy of face recognition from expressional face images.

Estimation of local orientations in multivariate signals is an important problem in image processing and computer vision. This general problem formulation also covers optical flow estimation, which can be regarded as orientation estimation in space-time-volumes. Modelling a signal using only a single orientation, however, is often too restrictive, since occlusions and transparencies occur frequently, thus necessitating the modelling and analysis of multiple orientations. We, therefore, develop a unifying mathematical model for multiple orientations: Beyond describing an arbitrary number of orientations in scalar- and vector-valued image data such as color image sequences, it allows the unified treatment of additively and occludingly superimposed oriented structures as well as of combinations of these. Based on this model, we describe estimation schemes for an arbitrary number of additively or occludingly superimposed orientations in images. We confirm the performance of our framework on both synthetic and real image data.

We present a variational framework for deinterlacing that was originally used for inpainting and subsequently redeveloped for deinterlacing. From the framework, we derive a motion adaptive (MA) deinterlacer and a motion compensated (MC) deinterlacer and test them together with a selection of known deinterlacers. To illustrate the need for MC deinterlacing, the problem of details in motion (DIM) is introduced. It cannot be solved by MA deinterlacers or any simpler deinterlacers but only by MC deinterlacers. The major problem in MC deinterlacing is computing reliable optical flow [motion estimation (ME)] in interlaced video. We discuss a number of strategies for computing optical flows on interlaced video hoping to shed some light on this problem. We produce results on challenging real world video data with our variational MC deinterlacer that in most cases are indistinguishable from the ground truth.

Optical flow can be reliably estimated between areas visible in two images, but not in occlusion areas. If optical flow is needed in the whole image domain, one approach is to use additional views of the same scene. If such views are unavailable, an often-used alternative is to extrapolate optical flow in occlusion areas. Since the location of such areas is usually unknown prior to optical flow estimation, this is usually performed in three steps. First, occlusion-ignorant optical flow is estimated, then occlusion areas are identified using the estimated (unreliable) optical flow, and, finally, the optical flow is corrected using the computed occlusion areas. This approach, however, does not permit interaction between optical flow and occlusion estimates. In this paper, we permit such interaction by proposing a variational formulation that jointly computes optical flow, implicitly detects occlusions and extrapolates optical flow in occlusion areas. The extrapolation mechanism is based on anisotropic diffusion and uses the underlying image gradient to preserve structure, such as optical flow discontinuities. Our results show significant improvements in the computed optical flow fields over other approaches, both qualitatively and quantitatively.

Robust estimation of the optical flow is addressed through a multiresolution energy minimization. It involves repeated evaluation of spatial and temporal gradients of image intensity which rely usually on bilinear interpolation and image filtering. We propose to base both computations on a single pyramidal cubic B-spline model of image intensity. We show empirically improvements in convergence speed and estimation error and validate the resulting algorithm on real test sequences

Geometrical distortions are the Achilles heel for many watermarking schemes. Most countermeasures proposed in the literature only address the problem of global affine transforms (e.g., rotation, scaling, and translation). In this paper, we propose an original blind watermarking algorithm robust to local geometrical distortions such as the deformations induced by Stirmark. Our method consists in adding a predefined additional information to the useful message bits at the insertion step. These additional bits are labeled as resynchronization bits or reference bits and they are modulated in the same way as the information bits. During the extraction step, the reference bits are used as anchor points to estimate and compensate for small local and global geometrical distortions. The deformations are approximated using a modified basic optical flow algorithm.

We present an approach to parallel variational optical-flow computation by using an arbitrary partition of the image plane and iteratively solving related local variational problems associated with each subdomain. The approach is particularly suited for implementations on PC clusters because interprocess communication is minimized by restricting the exchange of data to a lower dimensional interface. Our mathematical formulation supports various generalizations to linear/nonlinear convex variational approaches, three-dimensional image sequences, spatiotemporal regularization, and unstructured geometries and triangulations. Results concerning the effects of interface preconditioning, as well as runtime and communication volume measurements on a PC cluster, are presented. Our approach provides a major step toward real-time two-dimensional image processing using off-the-shelf PC hardware and facilitates the efficient application of variational approaches to large-scale image processing problems.

Recent advances in imaging sensor technology make high frame-rate video capture practical. As demonstrated in previous work, this capability can be used to enhance the performance of many image and video processing applications. The idea is to use the high frame-rate capability to temporally oversample the scene and, thus, to obtain more accurate information about scene motion and illumination. This information is then used to improve the performance of image and standard frame-rate video applications. This paper investigates the use of temporal oversampling to improve the accuracy of optical flow estimation (OFE). A method for obtaining high accuracy optical flow estimates at a conventional standard frame rate, e.g., 30 frames/s, by first capturing and processing a high frame-rate version of the video is presented. The method uses the Lucas-Kanade algorithm to obtain optical flow estimates at a high frame rate, which are then accumulated and refined to estimate the optical flow at the desired standard frame rate. The method demonstrates significant improvements in OFE accuracy both on synthetically generated video sequences and on a real video sequence captured using an experimental high-speed imaging system. It is then shown that a key benefit of using temporal oversampling to estimate optical flow is the reduction in motion aliasing. Using sinusoidal input sequences, the reduction in motion aliasing is identified and the desired minimum sampling rate as a function of the velocity and spatial bandwidth of the scene is determined. Using both synthetic and real video sequences, it is shown that temporal oversampling improves OFE accuracy by reducing motion aliasing not only for areas with large displacements but also for areas with small displacements and high spatial frequencies. The use of other OFE algorithms with temporally oversampled video is then discussed. In particular, the Haussecker algorithm is extended to work with high frame-rate sequences. This extension demonstrates yet another important benefit of temporal oversampling, which is improving OFE accuracy when brightness varies with time.

A new conditional formulation of classical filtering methods is proposed. This formulation is dedicated to image sequence-based tracking. These conditional filters allow solving systems whose measurements and state equation are estimated from the image data. In particular, the model that is considered for point tracking combines a state equation relying on the optical flow constraint and measurements provided by a matching technique. Based on this, two point trackers are derived. The first one is a linear tracker well suited to image sequences exhibiting global-dominant motion. This filter is determined through the use of a new estimator, called the conditional linear minimum variance estimator. The second one is a nonlinear tracker, implemented from a conditional particle filter. It allows tracking of points whose motion may be only locally described. These conditional trackers significantly improve results in some general situations. In particular, they allow for dealing with noisy sequences, abrupt changes of trajectories, occlusions, and cluttered background. Fil: Arnaud, Elise. Universite de Rennes I; Francia Fil: Memin, Etienne. Universite de Rennes I; Francia Fil: Cernuschi Frias, Bruno. Consejo Nacional de Investigaciones Científicas y Técnicas. Oficina de Coordinación Administrativa Saavedra 15. Instituto Argentino de Matemática Alberto Calderón; Argentina. Universidad de Buenos Aires. Facultad de Ingeniería; Argentina

The task of image registration is fundamental in image processing. It often is a critical preprocessing step to many modern image processing and computer vision tasks, and many algorithms and techniques have been proposed to address the registration problem. Often, the performances of these techniques have been presented using a variety of relative measures comparing different estimators, leaving open the critical question of overall optimality. In this paper, we present the fundamental performance limits for the problem of image registration as derived from the Cramer-Rao inequality. We compare the experimental performance of several popular methods with respect to this performance bound, and explain the fundamental tradeoff between variance and bias inherent to the problem of image registration. In particular, we derive and explore the bias of the popular gradient-based estimator showing how widely used multiscale methods for improving performance can be explained with this bias expression. Finally, we present experimental simulations showing the general rule-of-thumb performance limits for gradient-based image registration techniques.

Motion transparency phenomena in image sequences are frequent, but classical methods of motion estimation are unable to deal with them. This paper describes a method for estimating optical flow by a generalization of the brightness constancy assumption to additive transparencies. The brightness constancy assumption is obtained by setting constant velocity fields during three images of a sequence. Thus, by a Taylor development to its second order, we reach an extension of the optical flow constraint equation. Since the equation is nonlinear, the Levenberg-Marquardt algorithm is used. In order to suppress the unavoidable aperture problem, a global model based on B-spline basis functions is applied with the aim of constraining optical flows. This description of motion allows us to work on a coarse to fine estimation of artificial image sequences that shows good convergence properties. It is also applied to natural image sequences.

The goal of three-dimensional (3-D) motion segmentation is to identify the image areas projected by different moving objects in 3-D space. However, many prevailing methods merely detected the discontinuity of optical flow field and usually considered these boundaries as that produced by different 3-D motions. In fact, the flow discontinuity can be generated either by two different 3-D motions or by the structural discontinuity on the same moving object. The wrong identification causes several problems in 3-D motion estimation. A simple method called the extrapolation and subtraction (ES) technique is proposed to solve these problems. The input image flow field is first partitioned into several functionally analytic regions. Each analytic region is assumed to be projected by a roughly planar patch moving in 3-D space. Based on the parameterization of these analytic flow fields, the ES technique provides a very simple and fast method to test the 3-D motion compatibility between two interested analytic flow fields.

The issues governing the computation of optical flow in image sequences are addressed. The trade-off between accuracy versus computation cost is shown to be dependent on the redundancy of the image representation. This dependency is highlighted by reformulating Horn's (1986) algorithm, making explicit use of the approximations to the continuous basis functions underlying the discrete representation. The computation cost of estimating optical flow, for a fixed error tolerance, is shown to be a minimum for images resampled at twice the Nyquist rate. The issues of derivative calculation and multiresolution representation are also briefly discussed in terms of basis functions and information encoding. A multiresolution basis function formulation of Horn's algorithm is shown to lead to large improvements in dealing with high frequencies and large displacements.

Several approaches for optical flow estimation use partial differential equations to model changes in image brightness throughout time. A commonly used equation is the so-called optical flow constraint (OFC), which assumes that the image brightness is stationary with respect to time. More recently, a different constraint referred to as the extended optical flow constraint (EOFC) has been introduced, which also contains the divergence of the flow field of image brightness. There is no agreement in the literature about which of these constraints provides the best estimation of the velocity field. Two new solutions for optical flow computation are proposed, which are based on an approximation of the constraint equations. The two techniques have been used with both EOFC and OFC constraint equations. Results achieved by using these solutions have been compared with several well-known computational methods for optical flow estimation in different motion conditions. Estimation errors have also been measured and compared for different types of motion.

Different constraint equations have been proposed in the literature for the derivation of optical flow. Despite of the large number of papers dealing with computational techniques to estimate optical flow, only a few authors have investigated conditions under which these constraints exactly model the velocity field, that is, the perspective projection on the image plane of the true 3-D velocity. These conditions are analyzed under different hypotheses, and the departures of the constraint equations in modeling the velocity field are derived for different motion conditions. Experiments are also presented giving measures of these departures and of the induced errors in the estimation of the velocity field. >