Search

Advancements in deep neural networks (DNNs) and widespread deployment of video cameras have fueled the need for video analytics systems. Despite rapid advances in system design, existing systems treat DNNs largely as "black boxes'' and either deploy models entirely on a camera or compress videos for analysis in the cloud. Both these approaches affect the accuracy and total cost of deployment. In this position paper, we propose a research agenda that involves opening up the black box of neural networks and describe new application scenarios that include joint inference between the cameras and the cloud, and continuous online learning for large deployments of cameras. We present promising results from preliminary work in efficiently encoding the intermediate activations sent between layers of a neural network and describe opportunities for further research.

We describe a behavioral navigation approach that leverages the rich semantic structure of human environments to enable robots to navigate without an explicit geometric representation of the world. Based on this approach, we then present our efforts to allow robots to follow navigation instructions in natural language. With our proof-of-concept implementation, we were able to translate natural language navigation commands into a sequence of behaviors that could then be executed by a robot to reach a desired goal.

This paper deals with the problem of computing a semantic segmentation of an image via label transfer from an already labeled image set. In particular it proposes a method that takes advantage of sparse 3D structure to infer the category of superpixel in the novel image. The label assignment is computed by a Markov random field that has the superpixels of the image as nodes. The data term combines labeling proposals from the appearance of the superpixel and from the 3D structure, while the pairwise term incorporates spatial context, both in the image and in 3D space. Exploratory results indicate that 3D structure, albeit sparse, improves the process of label transfer.

The deployment of computer vision algorithms in mobile applications is growing at a rapid pace. A primary component of the computer vision software pipeline is feature extraction, which identifies and encodes relevant image features. We present an embedded heterogeneous multicore design named EFFEX that incorporates novel functional units and memory architecture support, making it capable of increasing mobile vision performance while balancing power and area. We demonstrate this architecture running three common feature extraction algorithms, and show that it is capable of providing significant speedups at low cost. Our simulations show a speedup of as much as 14× for feature extraction with a decrease in energy of 40× for memory accesses.

Three-dimensional shape may be perceived from static images. Contours, shading, texture gradients, perspective and occlusion are well-studied cues to this percept. When looking at a picture of a specular object, such as a silver vase, one additional cue is potentially available: a deformed picture of the reflected environment is seen in the surface of the object and the amount and type of deformation depend on its shape. Can specular reflections be used as a visual cue for shape perception? Our experiments show that our subjects are very poor at judging the shape of mirror surfaces in absence of other visual cues. However, for a considerable subset of the stimuli, subjects are highly consistent in their (most often wrong) perception. This observation leads us to the hypothesis that our subjects rather than 'computing' a percept from each image based on geometrical considerations, may be associating a shape to each pattern in a stereotypical way, akin to pattern-matching. This behavior is reasonable since, as suggested by our ideal observer analysis, the information available from specular reflections is ambiguous when the surrounding world is (partially) unknown.