Search

Face recognition performance evaluation has traditionally focused on one-to-one verification, popularized by the Labeled Faces in the Wild data set [1] for imagery and the YouTubeFaces data set [2] for videos. In contrast, the newly released IJB-A face recognition data set [3] unifies evaluation of one-to-many face identification with one-to-one face verification over templates, or sets of imagery and videos for a subject. In this paper, we study the problem of template adaptation, a form of transfer learning to the set of media in a template. Extensive performance evaluations on IJB-A show a surprising result, that perhaps the simplest method of template adaptation, combining deep convolutional network features with template specific linear SVMs, outperforms the state-of-the-art by a wide margin. We study the effects of template size, negative set construction and classifier fusion on performance, then compare template adaptation to convolutional networks with metric learning, 2D and 3D alignment. Our unexpected conclusion is that these other methods, when combined with template adaptation, all achieve nearly the same top performance on IJB-A for template-based face verification and identification.

We present an end-to-end system for detecting and clustering faces by identity in full-length movies. Unlike works that start with a predefined set of detected faces, we consider the end-to-end problem of detection and clustering together. We make three separate contributions. First, we combine a state-of-the-art face detector with a generic tracker to extract high quality face tracklets. We then introduce a novel clustering method, motivated by the classic graph theory results of Erdos and Renyi. It is based on the observations that large clusters can be fully connected by joining just a small fraction of their point pairs, while just a single connection between two different people can lead to poor clustering results. This suggests clustering using a verification system with very few false positives but perhaps moderate recall. We introduce a novel verification method, rank-1 counts verification, that has this property, and use it in a link-based clustering scheme. Finally, we define a novel end-to-end detection and clustering evaluation metric allowing us to assess the accuracy of the entire end-to-end system. We present state-of-the-art results on multiple video data sets and also on standard face databases.

Densely and regularly sampled ground-level imagery collected from semi-automated vehicle collection platforms (e.g. Street ViewTM or EarthMineTM) is rapidly becoming available on a global scale. Ground level views of neighborhoods around the world can provide unique regional indicators (e.g. demographics, zoning, socioeconomic health) at a finer resolution than current census data. To this end, this paper introduces the first unsupervised method for automated hierarchical modeling of high-level latent regions using densely sampled, geotagged ground imagery at a global scale. Our unsupervised, nonparametric approach models a multi-scale regularly sampled grid of panoramic ground-level images (where available), producing a multi-scale, multivariate estimate of the latent types of locations (e.g. commercial street, wooded area, park, country road, suburban street, etc.). Region types are model distributions of location types characteristic of particular neighborhoods, towns, or cities. These latent region types are used to cluster all locations into distinct geographic entities with regional signatures that can be used for comparison. We show the effectiveness of this method for discovering regional distributions at vastly different scales, including the Boston area, the United States, and the World.

SeeCoast is a prototype US Coast Guard port and coastal area surveillance system that aims to reduce operator workload while maintaining optimal domain awareness by shifting their focus from having to detect events to being able to analyze and act upon the knowledge derived from automatically detected anomalous activities. The automated scene understanding capability provided by the baseline SeeCoast system (as currently installed at the Joint Harbor Operations Center at Hampton Roads, VA) results from the integration of several components. Machine vision technology processes the real-time video streams provided by USCG cameras to generate vessel track and classification (based on vessel length) information. A multi-INT fusion component generates a single, coherent track picture by combining information available from the video processor with that from surface surveillance radars and AIS reports. Based on this track picture, vessel activity is analyzed by SeeCoast to detect user-defined unsafe, illegal, and threatening vessel activities using a rule-based pattern recognizer and to detect anomalous vessel activities on the basis of automatically learned behavior normalcy models. Operators can optionally guide the learning system in the form of examples and counter-examples of activities of interest, and refine the performance of the learning system by confirming alerts or indicating examples of false alarms. The fused track picture also provides a basis for automated control and tasking of cameras to detect vessels in motion. Real-time visualization combining the products of all SeeCoast components in a common operating picture is provided by a thin web-based client.

SeeCoast extends the US Coast Guard Port Security and Monitoring system by adding capabilities to detect, classify, and track vessels using electro-optic and infrared cameras, and also uses learned normalcy models of vessel activities in order to generate alert cues for the watch-standers when anomalous behaviors occur. SeeCoast fuses the video data with radar detections and Automatic Identification System (AIS) transponder data in order to generate composite fused tracks for vessels approaching the port, as well as for vessels already in the port. Then, SeeCoast applies rule-based and learning-based pattern recognition algorithms to alert the watch-standers to unsafe, illegal, threatening, and other anomalous vessel activities. The prototype SeeCoast system has been deployed to Coast Guard sites in Virginia. This paper provides an overview of the system and outlines the lessons learned to date in applying data fusion and automated pattern recognition technology to the port security domain.

Computer vision is the most promising technology for automated, passive tracking of multiple objects over large areas. Effecti ve inter-camera and intra-camera visual tracking can enable information about a vehicle or a pedestrian to be integrated from various sources. Unfortunately, tracking objects across multiple non-overlapping cameras requires reliable comparison of the objects’ appearance under widely-varying view an gles and resolutions. Fortunately, in most cases, an object of a particul ar type entering a scene at a particular position and direction will tend to be i n a very similar pose. This paper introduces Translation Templates (TTs). TTs exploit this regularity to learn a color-based matching metric for images from a pair of tracking source and sink points, without prior knowledge of object type or object pose. This model benefits from histogram-based aggre gation while still preserving spatial relationships between the two ima ges. The model can be learned directly from data and used to compare arbitrary types of objects observed from extremely different viewpoints, as long as the relationship between the viewpoints is preserved. This paper describes TTs, describes a method for efficient computation and for visualization of TT s, and presents experimental results from both an indoor pedestrian data set and an outdoor vehicle data set.

This paper investigates applications of a new representation for images, the similarity template. A similarity template is a probabilistic representation of the similarity of pixels in an image patch. It has application to detection of a class of objects, because it is reasonably invariant to the color of a particular object. Further, it enables the decomposition of a class of objects into component parts over which robust statistics of color can be approximated. These regions can be used to create a factored color model that is useful for recognition. Detection results are shown on a system that learns to detect a class of objects (pedestrians) in static scenes based on examples of the object provided automatically by a tracking system. Applications of the factored color model to image indexing and anomaly detection are pursued on a database of images of pedestrians.

There are many methods for measuring "similarity" of or "distance" between two image pixels or image patches. These methods generally involve computing some informative features of the image patches and then describing distances between patches based on simple functions of those feature values, e.g., Euclidean distance. Unfortunately, these measures are often frail and difficult to interpret. The goal of this paper is to learn the "similarity" of two patches as an approximation to the likelihood that two patches were drawn from the same surface in the world. This measure is well-defined and allows for a maximization of meaningful values when it is combined with common segmentation algorithms. We introduce a general approximation technique that involves learning a codebook, learning the likelihood function for pairs of codebook entries, and applying the resulting likelihood function in segmentation tasks. These steps can be performed independently, even on different data sets drawn from the same domain. The likelihood can be learned from pre-segmented image set or heuristically approximated from an unsegmented image set. We show examples of probabilistic segmentation of the codebooks themselves based on our similarity measure for multiple types of codebooks including color codebooks and Epitome codebooks. These segmentations illustrate the usefulness of our technique as an image patch similarity measure.

There are many useful observable characteristics of the state of a tracked object. These characteristics could include normalized size, normalized speed, normalized direction, object color, position, and object shape among other characteristics. Although these characteristics are by no means completely independent of each other, it is desirable to determine a separate, compact description of each of each of these aspects. Using this compact factored description, different aspects of individual sequences can be estimated and described without overwhelming computational or storage costs. In this work, we describe Factored Latent Analysis (FLA) and its application to deriving factored models for segmenting sequences in each of K separate characteristics. This method exploits temporally local statistics within each of the latent aspects and their inter-dependencies to derive a model that allows segmentation of each of the observed characteristics. This method is data driven and unsupervised. Activity classification results for multiple challenging environments are shown.

This paper describes an automatic surveillance system, which performs labeling of events and interactions in an outdoor environment. The system is designed to monitor activities in an open parking lot. It consists of three components-an adaptive tracker, an event generator, which maps object tracks onto a set of pre-determined discrete events, and a stochastic parser. The system performs segmentation and labeling of surveillance video of a parking lot and identifies person-vehicle interactions, such as pick-up and drop-off. The system presented in this paper is developed jointly by MIT Media Lab and MIT Artificial Intelligence Lab.

While a tracking system is unaware of the identity of any object it tracks, the identity remains the same for the entire tracking sequence. Our system leverages this information by using accumulated joint cooccurrences of the representations within the sequence to create a hierarchical binary-tree classifier of the representations. This classifier is useful to classify sequences as well as individual instances. We illustrate the use of this method on two separate representations the tracked object's position, movement, and size; and the tracked object's binary motion silhouettes.

A common method for real-time segmentation of moving regions in image sequences involves "background subtraction", or thresholding the error between an estimate of the image without moving objects and the current image. The numerous approaches to this problem differ in the type of background model used and the procedure used to update the model. This paper discusses modeling each pixel as a mixture of Gaussians and using an on-line approximation to update the model. The Gaussian, distributions of the adaptive mixture model are then evaluated to determine which are most likely to result from a background process. Each pixel is classified based on whether the Gaussian distribution which represents it most effectively is considered part of the background model. This results in a stable, real-time outdoor tracker which reliably deals with lighting changes, repetitive motions from clutter, and long-term scene changes. This system has been run almost continuously for 16 months, 24 hours a day, through rain and snow.

We describe a vision system that monitors activity in a site over extended periods of time. The system uses a distributed set of sensors to cover the site, and an adaptive tracker detects multiple moving objects in the sensors. Our hypothesis is that motion tracking is sufficient to support a range of computations about site activities. We demonstrate using the tracked motion data to calibrate the distributed sensors, to construct rough site models, to classify detected objects, to learn common patterns of activity for different object classes, and to detect unusual activities.