Search

We experimentally demonstrate that an uncertainty-aware framework for robotic navigation and vision is able to navigate a robotic platform around an obstacle without choosing an overly long and conservative path. The framework contains many interconnected experimental pieces, including monocular visual-inertial odometry (VIO) based on an Extended Kalman Filter (EKF), a recurrent neural network that predicts future covariance estimates from the EKF, a model predictive controller that uses uncertainty in its cost function, and an object detection network. Each interconnected piece is an algorithm that makes assumptions about its inputs, which are the outputs of another piece. The rest of the thesis is a systems validation exercise that examines several of these assumptions for validity and finds that they are largely not true. First, we learn that uncertainty estimates of the commonly used EKF are overconfident, but that the overconfidence is systematic and correctable. Next, we examine the distribution of feature track errors and find that not only are the errors not zero-mean Gaussian, the errors are dependent on motion type, speed, and the type of feature tracking algorithm used. We then quantify the effect of attribution errors, Gaussian noise, and drift on performance and uncertainty estimates of the VIO algorithm used in the framework. Finally, we attempt to characterize the uncertainty of image classification networks in a manner appropriate for online navigation. To our knowledge, the proposed architecture is new and this dissertation is the first time a systems validation exercise has focused on uncertainty estimation.

This dissertation provides a generic solution to model dynamic systems whose hidden state and the transition model are unknown in practice. We build the task-sufficient filtering framework to maintain a finite, abstract, and learnable representation (memory) that is sufficient to update itself, casually and iteratively, and to predict downstream task variables of interest. We show our realization of the framework by recurrent neural networks as universally-approximating function classes to imitate the functionality of a state transition model and a task prediction model.In addition, we provide practical methodologies to impose generic priors of the physical scene on the hidden representation. We leverage (lower-level) topological and regularity constraints of natural images, such as occlusion relations, to define object regions. Hence, we capture the motion priors associated with different (higher-level) semantic categories, that are combined to describe the dynamics of the whole scene.The framework takes videos as sequential input streams and produces representations of video dynamics. We show the success of our framework by applying it to solve real-world computer vision tasks, including generic object tracking and video prediction. The learned dynamic models are extensible to multiple circumstances requiring a dynamically and casually updated memory with uncertainty.

This work studies the effect of adversarial perturbations of images on the estimates of disparity by deep learning models trained for stereo. We show that imperceptible additive perturbations can significantly alter the disparity map, and correspondingly the perceived geometry of the scene. These perturbations not only affect the specific model they are crafted for, but transfer to models with different architecture, trained with different loss functions. We show that, when used for adversarial data augmentation, our perturbations result in trained models that are more robust, without sacrificing overall accuracy of the model. This is unlike what has been observed in image classification, where adding the perturbed images to the training set makes the model less vulnerable to adversarial perturbations, but to the detriment of overall accuracy. We test our method using the most recent stereo networks and evaluate their performance on public benchmark datasets.

We focus on two broad learning setups: The first one is the classic semi-supervised learning (SSL), wherein few labeled samples and many unlabeled samples are drawn from the same distribution. Second is a more challenging setting called unsupervised domain adaptation (UDA), where labeled and unlabeled samples are drawn from slightly different distributions. This setting is more practical as labeled data can be drawn from synthetic data, which can be produced abundantly using graph engines with the expense of domain shift from the real data. Our contributions are multi-faceted. For the SSL setup, we show that the training speed in the supervised setting correlates strongly with the percentage of correct labels. Since the speed of the training is a measurable quantity at the training time unlike the accuracy of the estimates, we propose to use it as an inference criterion. We additionally show that robustness to small perturbations in input space and weight space do not imply each other and they both improve generalization performance. Then, we propose a method that combines an input-space smoothing with a weight-space smoothing. In the UDA setup, we begin by proposing a method that trains a shared embedding to align the distributions of the learned features conditioned on the classes. To have more interpretable models, we also explore the use of generative modeling where we generate images in the unlabeled target domain in a manner that allows independent control of class and nuisance variability. We also study UDA for dense prediction tasks like semantic segmentation where the manual annotation is more costly. Lastly, we examine monocular depth prediction tasks where the goal is to infer dense depth maps from images. Obtaining the 3D scene from a single image is a degenerate task as there are infinitely many 3D scenes compatible with the given image. So, we also leverage LiDAR measurements, which brings the additional challenges of processing sparse inputs. We propo

We focus on two broad learning setups: The first one is the classic semi-supervised learning (SSL), wherein few labeled samples and many unlabeled samples are drawn from the same distribution. Second is a more challenging setting called unsupervised domain adaptation (UDA), where labeled and unlabeled samples are drawn from slightly different distributions. This setting is more practical as labeled data can be drawn from synthetic data, which can be produced abundantly using graph engines with the expense of domain shift from the real data. Our contributions are multi-faceted. For the SSL setup, we show that the training speed in the supervised setting correlates strongly with the percentage of correct labels. Since the speed of the training is a measurable quantity at the training time unlike the accuracy of the estimates, we propose to use it as an inference criterion. We additionally show that robustness to small perturbations in input space and weight space do not imply each other and they both improve generalization performance. Then, we propose a method that combines an input-space smoothing with a weight-space smoothing. In the UDA setup, we begin by proposing a method that trains a shared embedding to align the distributions of the learned features conditioned on the classes. To have more interpretable models, we also explore the use of generative modeling where we generate images in the unlabeled target domain in a manner that allows independent control of class and nuisance variability. We also study UDA for dense prediction tasks like semantic segmentation where the manual annotation is more costly. Lastly, we examine monocular depth prediction tasks where the goal is to infer dense depth maps from images. Obtaining the 3D scene from a single image is a degenerate task as there are infinitely many 3D scenes compatible with the given image. So, we also leverage LiDAR measurements, which brings the additional challenges of processing sparse inputs. We propo

This work studies the effect of adversarial perturbations of images on the estimates of disparity by deep learning models trained for stereo. We show that imperceptible additive perturbations can significantly alter the disparity map, and correspondingly the perceived geometry of the scene. These perturbations not only affect the specific model they are crafted for, but transfer to models with different architecture, trained with different loss functions. We show that, when used for adversarial data augmentation, our perturbations result in trained models that are more robust, without sacrificing overall accuracy of the model. This is unlike what has been observed in image classification, where adding the perturbed images to the training set makes the model less vulnerable to adversarial perturbations, but to the detriment of overall accuracy. We test our method using the most recent stereo networks and evaluate their performance on public benchmark datasets.

We describe components of a visual perception system to understand the geometry and semantics of the three-dimensional scene by utilizing monocular cameras and inertial measurement units (IMUs). The use of the two sensor modalities is motivated by the wide availability of the camera-IMU sensor packages present in mobile devices from phones to cars, and their complementary sensing capabilities: IMUs can track the motion of the sensor platform over a short period of time accurately, and provide a scaled and gravity-aligned global reference frame, while cameras can capture rich photometric signatures of the scene, and provide relative motion constraints between images up to scale. We first show that visual 3D reconstruction can be improved by leveraging the global orientation frame -- easily inferred from inertials. In the gravity-aligned global orientation frame, a shape prior can be imposed in depth prediction from a single image, where the normal vectors to surfaces of objects of certain classes tend to align with gravity or orthogonal to it. Adding such a prior to baseline methods for monocular depth prediction yields improvements beyond the state-of-the-art and illustrates the power of utilizing inertials in 3D reconstruction. The global reference provided by inertials is not only gravity-aligned but also scaled, which is exploited in depth completion: We describe a method to infer dense metric depth from camera motion and sparse depth as estimated using a visual-inertial odometry system. Unlike other scenarios using point clouds from lidar or structured light sensors, we have few hundreds to few thousand points, insufficient to inform the topology of the scene. Our method first constructs a piecewise planar scaffolding of the scene, and then uses it to infer dense depth using the image along with the sparse points. We use a predictive cross-modal criterion, akin to “self-supervision,” measuring photometric consistency across time, forward-backward pose consistenc

Recovering three-dimensional (3D) scene geometry from images is an ill-posed problem due to the loss of the extra dimension in the process of projection. Hence, the solution hinges on the choice of regularization or prior assumptions about the scene. We study the effects of various regularization schemes on the 3D reconstruction problem under two problem settings, single image depth prediction and sparse depth completion. Obtaining the 3D scene from a single image is literally an impossible task as there are infinitely many 3D scenes compatible with the given image -- making both problem settings great candidates for evaluating the influence of a given regularizer. We begin by examining the relation between data fidelity residual and the degree of regularization to form a spatially and temporally varying adaptive weighting scheme for single image depth prediction. We additionally explore the use of gravity, as a supervisory signal, to induce a prior on the pose of objects populating a scene. To extend the use case to real world applications, we develop visioning systems to infer dense depth from an image with associated sparse depth measurements. We leverage the abundance of synthetic data to obtain a learned prior for guiding the learning process. Conscious of the limitations of current depth completion methods in processing sparse depth and their growth in parameters, we propose a two-stage approach that approximates the scene with a ``scaffolding'' and refines the approximation with a simple light-weight network. The result is a small and fast, but accurate visioning system that fits in an embedded system. To enable an agent to continuously learn, our systems are completely unsupervised and learn by exploiting geometry and known regularities.

We show that information theoretic quantities can be used to control and describe the training process of Deep Neural Networks, and can explain how properties, such as invariance to nuisance variability and disentanglement of semantic factors, emerge naturally in the learned representation. Through its dynamics, stochastic gradient descent (SGD) implicitly regularizes the information in the weights, which can then be used to bound the generalization error through the PAC-Bayes bound. Moreover, the information in the weights can be used to defined both a topology and an asymmetric distance in the space of tasks, which can then be used to predict the training time and the performance on a new task given a solution to a pre-training task. While this information distance models difficulty of transfer in first approximation, we show the existence of non-trivial irreversible dynamics during the initial transient phase of convergence when the network is acquiring information, which makes the approximation fail. This is closely related to critical learning periods in biology, and suggests that studying the initial convergence transient can yield important insight beyond those that can be gleaned from the well-studied asymptotics.

The power of deep neural networks comes mainly from huge labeled datasets. Even though it shines on many computer vision tasks, supervised learning bears little hope to hack into the core of intelligent visual systems. On the other side, unsupervised learning is believed to be the future of AI; however, its performance is always inferior compared to the supervised counterpart. The goal of our research is to develop unsupervised learning algorithms for computer vision tasks while matching or even outperforming the supervised ones. Our key is a representation that is as informative as the supervisory labels, which can be constructed from an unlimited amount of unlabeled data. In theory, this representation contains richer information than the processed supervisory signal. Moreover, we develop algorithms that can utilize existing labeled datasets to expedite the information extraction from the unlimited unlabeled data. Our research is lined up in an order similar to the visual development in early infancy, such that we can also investigate the interplay between different visual functionalities. The final goal is to develop a robotic visual system akin to a human's, that can automatically acquire semantics from concepts of objects fostered by basic perceptions of motion and depth with the minimum amount of human supervision.

The power of deep neural networks comes mainly from huge labeled datasets. Even though it shines on many computer vision tasks, supervised learning bears little hope to hack into the core of intelligent visual systems. On the other side, unsupervised learning is believed to be the future of AI; however, its performance is always inferior compared to the supervised counterpart. The goal of our research is to develop unsupervised learning algorithms for computer vision tasks while matching or even outperforming the supervised ones. Our key is a representation that is as informative as the supervisory labels, which can be constructed from an unlimited amount of unlabeled data. In theory, this representation contains richer information than the processed supervisory signal. Moreover, we develop algorithms that can utilize existing labeled datasets to expedite the information extraction from the unlimited unlabeled data. Our research is lined up in an order similar to the visual development in early infancy, such that we can also investigate the interplay between different visual functionalities. The final goal is to develop a robotic visual system akin to a human's, that can automatically acquire semantics from concepts of objects fostered by basic perceptions of motion and depth with the minimum amount of human supervision.

We show that information theoretic quantities can be used to control and describe the training process of Deep Neural Networks, and can explain how properties, such as invariance to nuisance variability and disentanglement of semantic factors, emerge naturally in the learned representation. Through its dynamics, stochastic gradient descent (SGD) implicitly regularizes the information in the weights, which can then be used to bound the generalization error through the PAC-Bayes bound. Moreover, the information in the weights can be used to defined both a topology and an asymmetric distance in the space of tasks, which can then be used to predict the training time and the performance on a new task given a solution to a pre-training task. While this information distance models difficulty of transfer in first approximation, we show the existence of non-trivial irreversible dynamics during the initial transient phase of convergence when the network is acquiring information, which makes the approximation fail. This is closely related to critical learning periods in biology, and suggests that studying the initial convergence transient can yield important insight beyond those that can be gleaned from the well-studied asymptotics.

The power of deep neural networks comes mainly from huge labeled datasets. Even though it shines on many computer vision tasks, supervised learning bears little hope to hack into the core of intelligent visual systems. On the other side, unsupervised learning is believed to be the future of AI; however, its performance is always inferior compared to the supervised counterpart. The goal of our research is to develop unsupervised learning algorithms for computer vision tasks while matching or even outperforming the supervised ones. Our key is a representation that is as informative as the supervisory labels, which can be constructed from an unlimited amount of unlabeled data. In theory, this representation contains richer information than the processed supervisory signal. Moreover, we develop algorithms that can utilize existing labeled datasets to expedite the information extraction from the unlimited unlabeled data. Our research is lined up in an order similar to the visual development in early infancy, such that we can also investigate the interplay between different visual functionalities. The final goal is to develop a robotic visual system akin to a human's, that can automatically acquire semantics from concepts of objects fostered by basic perceptions of motion and depth with the minimum amount of human supervision.

We describe components of a visual perception system to understand the geometry and semantics of the three-dimensional scene by utilizing monocular cameras and inertial measurement units (IMUs). The use of the two sensor modalities is motivated by the wide availability of the camera-IMU sensor packages present in mobile devices from phones to cars, and their complementary sensing capabilities: IMUs can track the motion of the sensor platform over a short period of time accurately, and provide a scaled and gravity-aligned global reference frame, while cameras can capture rich photometric signatures of the scene, and provide relative motion constraints between images up to scale. We first show that visual 3D reconstruction can be improved by leveraging the global orientation frame -- easily inferred from inertials. In the gravity-aligned global orientation frame, a shape prior can be imposed in depth prediction from a single image, where the normal vectors to surfaces of objects of certain classes tend to align with gravity or orthogonal to it. Adding such a prior to baseline methods for monocular depth prediction yields improvements beyond the state-of-the-art and illustrates the power of utilizing inertials in 3D reconstruction. The global reference provided by inertials is not only gravity-aligned but also scaled, which is exploited in depth completion: We describe a method to infer dense metric depth from camera motion and sparse depth as estimated using a visual-inertial odometry system. Unlike other scenarios using point clouds from lidar or structured light sensors, we have few hundreds to few thousand points, insufficient to inform the topology of the scene. Our method first constructs a piecewise planar scaffolding of the scene, and then uses it to infer dense depth using the image along with the sparse points. We use a predictive cross-modal criterion, akin to “self-supervision,” measuring photometric consistency across time, forward-backward pose consistenc

Recovering three-dimensional (3D) scene geometry from images is an ill-posed problem due to the loss of the extra dimension in the process of projection. Hence, the solution hinges on the choice of regularization or prior assumptions about the scene. We study the effects of various regularization schemes on the 3D reconstruction problem under two problem settings, single image depth prediction and sparse depth completion. Obtaining the 3D scene from a single image is literally an impossible task as there are infinitely many 3D scenes compatible with the given image -- making both problem settings great candidates for evaluating the influence of a given regularizer. We begin by examining the relation between data fidelity residual and the degree of regularization to form a spatially and temporally varying adaptive weighting scheme for single image depth prediction. We additionally explore the use of gravity, as a supervisory signal, to induce a prior on the pose of objects populating a scene. To extend the use case to real world applications, we develop visioning systems to infer dense depth from an image with associated sparse depth measurements. We leverage the abundance of synthetic data to obtain a learned prior for guiding the learning process. Conscious of the limitations of current depth completion methods in processing sparse depth and their growth in parameters, we propose a two-stage approach that approximates the scene with a ``scaffolding'' and refines the approximation with a simple light-weight network. The result is a small and fast, but accurate visioning system that fits in an embedded system. To enable an agent to continuously learn, our systems are completely unsupervised and learn by exploiting geometry and known regularities.

This thesis characterizes the training process of deep neural networks. We are driven by two apparent paradoxes. First, optimizing a non-convex function such as the loss function of a deep network should be extremely hard, yet rudimentary algorithms like stochastic gradient descent are phenomenally successful at this. Second, over-parametrized models are expected to perform poorly on new data, yet large deep networks with millions of parameters achieve spectacular generalization performance.We build upon tools from two main areas to make progress on these questions: statistical physics and a continuous-time point-of-view of optimization. The former has been popular in the study of machine learning in the past and has been rejuvenated in recent years due to the strong correlation of empirical properties of modern deep networks with existing, older analytical results. The latter, i.e., modeling stochastic first-order algorithms as continuous-time stochastic processes, gives access to powerful tools from the theory of partial differential equations, optimal transportation and non-equilibrium thermodynamics.The confluence of these ideas leads to fundamental theoretical insights that explain observed phenomena in deep learning as well as the development of state-of-the-art algorithms for training deep networks.

This thesis characterizes the training process of deep neural networks. We are driven by two apparent paradoxes. First, optimizing a non-convex function such as the loss function of a deep network should be extremely hard, yet rudimentary algorithms like stochastic gradient descent are phenomenally successful at this. Second, over-parametrized models are expected to perform poorly on new data, yet large deep networks with millions of parameters achieve spectacular generalization performance.We build upon tools from two main areas to make progress on these questions: statistical physics and a continuous-time point-of-view of optimization. The former has been popular in the study of machine learning in the past and has been rejuvenated in recent years due to the strong correlation of empirical properties of modern deep networks with existing, older analytical results. The latter, i.e., modeling stochastic first-order algorithms as continuous-time stochastic processes, gives access to powerful tools from the theory of partial differential equations, optimal transportation and non-equilibrium thermodynamics.The confluence of these ideas leads to fundamental theoretical insights that explain observed phenomena in deep learning as well as the development of state-of-the-art algorithms for training deep networks.

Convolutional neural networks (CNNs) have risen to be the de-facto paragon for detecting the presence of objects in a scene, as portrayed by an image. CNNs are described as being "approximately invariant" to nuisance transformations such as planar translation, both by virtue of their convolutional architecture and by virtue of their approximation properties that, given sufficient parameters and training data, could in principle yield discriminants that are insensitive to nuisance transformations of the data. The fact that contemporary deep convolutional architectures appear very effective in classifying images as containing a given object regardless of its position, scale, and aspect ratio in large-scale benchmarks suggests that the network can effectively manage such nuisance variability. We conduct an empirical study and show that, contrary to popular belief, at the current level of complexity of convolutional architectures and scale of the data sets used to train them, CNNs are not very effective at marginalizing nuisance variability.This discovery leaves researchers the choice of investing more effort in the design of models that are less sensitive to nuisances or designing better region proposal algorithms in an effort to predict where the objects of interest lie and center the model around these regions. In this thesis steps towards both directions are made. First, we introduce DSP-CNN, which deploys domain-size pooling in order to transform the neural networks to be scale invariant in the convolutional operator level. Second, motivated by our empirical analysis, we propose novel sampling and pruning techniques for region proposal schemes that improve the end-to-end performance in large-scale classification, detection and wide-baseline correspondence to state-of-the-art levels. Additionally,since a proposal algorithm involves the design of a classifier, whose results are to be fed to another classifier (a Category CNN), it seems natural to leverage on the latt

Estimating the optimal representation from sensor data has been one of the most challenging problems in computer vision research. Given a particular task, an optimal representation should contain the right information for answering queries related to the task. To be specific, such a representation should be a sufficient statistics of the data that is invariant to nuisance factors irrelevant to the task yet affecting the data. Among all the sufficient statistics, we desire the minimal that costs the least in terms of complexity. In terms of invariance, we want to achieve the maximal so that nuisance will not affect the inference at test time. In the first part of the dissertation, we show that it is possible to build such an optimal local descriptor that is a minimal sufficient statistic of the data and is maximally invariant to certain nuisance variables in the problem of establishing feature correspondence. Given only one single image, such nuisance group is quite restricted as a single view does not afford the ability to distinguish the intrinsic properties of the scene from the extrinsics. This restriction is lifted once multiple views of the same underlying scene become available. A theoretical framework is proposed to compute an optimal multiple-view local representation with view-point change-induced domain deformation marginalized. In the second part, we investigate the nuisance management ability of deep neural networks in the context of image classification and show that an explicit sampling-based marginalization technique can improve its performance significantly. This is in line with the principle developed in the previous part. Finally, we build a real-time system to estimate a visual-inertial-semantic representation of the 3D scene from both imaging and inertial measurements. Evidence from the imaging and inertial measurements are causally aggregated into the final estimate in a Bayesian filtering framework. The geometric and semantic properties of the

Convolutional neural networks (CNNs) have risen to be the de-facto paragon for detecting the presence of objects in a scene, as portrayed by an image. CNNs are described as being "approximately invariant" to nuisance transformations such as planar translation, both by virtue of their convolutional architecture and by virtue of their approximation properties that, given sufficient parameters and training data, could in principle yield discriminants that are insensitive to nuisance transformations of the data. The fact that contemporary deep convolutional architectures appear very effective in classifying images as containing a given object regardless of its position, scale, and aspect ratio in large-scale benchmarks suggests that the network can effectively manage such nuisance variability. We conduct an empirical study and show that, contrary to popular belief, at the current level of complexity of convolutional architectures and scale of the data sets used to train them, CNNs are not very effective at marginalizing nuisance variability.This discovery leaves researchers the choice of investing more effort in the design of models that are less sensitive to nuisances or designing better region proposal algorithms in an effort to predict where the objects of interest lie and center the model around these regions. In this thesis steps towards both directions are made. First, we introduce DSP-CNN, which deploys domain-size pooling in order to transform the neural networks to be scale invariant in the convolutional operator level. Second, motivated by our empirical analysis, we propose novel sampling and pruning techniques for region proposal schemes that improve the end-to-end performance in large-scale classification, detection and wide-baseline correspondence to state-of-the-art levels. Additionally,since a proposal algorithm involves the design of a classifier, whose results are to be fed to another classifier (a Category CNN), it seems natural to leverage on the latt

Estimating the optimal representation from sensor data has been one of the most challenging problems in computer vision research. Given a particular task, an optimal representation should contain the right information for answering queries related to the task. To be specific, such a representation should be a sufficient statistics of the data that is invariant to nuisance factors irrelevant to the task yet affecting the data. Among all the sufficient statistics, we desire the minimal that costs the least in terms of complexity. In terms of invariance, we want to achieve the maximal so that nuisance will not affect the inference at test time. In the first part of the dissertation, we show that it is possible to build such an optimal local descriptor that is a minimal sufficient statistic of the data and is maximally invariant to certain nuisance variables in the problem of establishing feature correspondence. Given only one single image, such nuisance group is quite restricted as a single view does not afford the ability to distinguish the intrinsic properties of the scene from the extrinsics. This restriction is lifted once multiple views of the same underlying scene become available. A theoretical framework is proposed to compute an optimal multiple-view local representation with view-point change-induced domain deformation marginalized. In the second part, we investigate the nuisance management ability of deep neural networks in the context of image classification and show that an explicit sampling-based marginalization technique can improve its performance significantly. This is in line with the principle developed in the previous part. Finally, we build a real-time system to estimate a visual-inertial-semantic representation of the 3D scene from both imaging and inertial measurements. Evidence from the imaging and inertial measurements are causally aggregated into the final estimate in a Bayesian filtering framework. The geometric and semantic properties of the

In spite of extensive consumer interest in domains such as autonomous driving, general purpose visual perception for autonomy remains a challenging problem. Inertial sensors such as gyroscopes and accelerometers, commonplace on smartphones today, offer complementary capabilities to visual sensors, and robotic systems typically fuse information from both visual and inertial sensors to further enhance their capabilities. In this thesis, we explore models and algorithms for constructing a representation from the fusion of visual and inertial data with the goal of supporting autonomy. We first analyze the observability properties of the standard model for visual-inertial sensor fusion, which combines Structure-from-Motion (SFM, or Visual SLAM) with inertial navigation, and overturn the common consensus that the model is observable. Informed by this analysis we develop robust inference techniques to enable our real-time visual-inertial navigation system implementation to achieve state-of-the-art relative motion estimation performance in challenging and dynamic environments. From the information provided by this process, we construct a representation of the agent's environment to act as a map that significantly improves location search time relative to standard methods, enabling long-term consistent localization within a constrained environment in real-time using commodity computing hardware. Finally, to allow an autonomous agent to reason about its world at a granularity relevant for interaction, we construct a dense surface representation upon this consistent map and develop algorithms to segment the surfaces into objects of potential relevance to the agent's task.

This thesis describes a framework leveraging occlusions as a cue for detecting objects and accurately localizing their boundaries throughout the course of a video. Triggered by the motion of objects in the scene, occlusions provide coarse knowledge of the spatial relationship of objects with respect to the viewer. While effective for detecting objects when motion is sufficient, we explore ways to reliably detect and track objects when motion is inadequate or difficult to estimate.In the first half, we incorporate semantic classifiers to provide cues when occlusions are weak, and observe occlusion and appearance information to be mutually beneficial, yielding results more resilient to failures of the component systems acting alone. Our system is evaluated on the semantic segmentation task. In the latter half, we drop semantics and instead devise a causal framework integrating segmentation results and occlusion cues from frames processed in the past. So long as objects move sufficiently with respect to the viewer at some point, they will be detected and subsequently tracked for the rest of the video. We evaluated our approach on the video object segmentation problem. The resulting system has the capability to automatically discover objects from occlusions in video and track their shapes as they evolve over time. Coarse depth is provided as a byproduct and the assignment of semantic category labels can be integrated in a natural way.

In spite of extensive consumer interest in domains such as autonomous driving, general purpose visual perception for autonomy remains a challenging problem. Inertial sensors such as gyroscopes and accelerometers, commonplace on smartphones today, offer complementary capabilities to visual sensors, and robotic systems typically fuse information from both visual and inertial sensors to further enhance their capabilities. In this thesis, we explore models and algorithms for constructing a representation from the fusion of visual and inertial data with the goal of supporting autonomy. We first analyze the observability properties of the standard model for visual-inertial sensor fusion, which combines Structure-from-Motion (SFM, or Visual SLAM) with inertial navigation, and overturn the common consensus that the model is observable. Informed by this analysis we develop robust inference techniques to enable our real-time visual-inertial navigation system implementation to achieve state-of-the-art relative motion estimation performance in challenging and dynamic environments. From the information provided by this process, we construct a representation of the agent's environment to act as a map that significantly improves location search time relative to standard methods, enabling long-term consistent localization within a constrained environment in real-time using commodity computing hardware. Finally, to allow an autonomous agent to reason about its world at a granularity relevant for interaction, we construct a dense surface representation upon this consistent map and develop algorithms to segment the surfaces into objects of potential relevance to the agent's task.

This thesis describes a framework leveraging occlusions as a cue for detecting objects and accurately localizing their boundaries throughout the course of a video. Triggered by the motion of objects in the scene, occlusions provide coarse knowledge of the spatial relationship of objects with respect to the viewer. While effective for detecting objects when motion is sufficient, we explore ways to reliably detect and track objects when motion is inadequate or difficult to estimate.In the first half, we incorporate semantic classifiers to provide cues when occlusions are weak, and observe occlusion and appearance information to be mutually beneficial, yielding results more resilient to failures of the component systems acting alone. Our system is evaluated on the semantic segmentation task. In the latter half, we drop semantics and instead devise a causal framework integrating segmentation results and occlusion cues from frames processed in the past. So long as objects move sufficiently with respect to the viewer at some point, they will be detected and subsequently tracked for the rest of the video. We evaluated our approach on the video object segmentation problem. The resulting system has the capability to automatically discover objects from occlusions in video and track their shapes as they evolve over time. Coarse depth is provided as a byproduct and the assignment of semantic category labels can be integrated in a natural way.

This thesis describes a research program towards enabling mobile visual agents to successfully solve visual decision problems. We identify perception, information acquisition, and prediction as the three areas to focus on, and develop methods to facilitate these stages, which are then applied to a sample selection of visual tasks. We first focus on segmentation and categorization of objects in video, and propose techniques for doing so. We then describe approaches for information-acquisition, useful whenever the data collection process can be controlled, or when one can choose which data to process. Finally, we describe an approach that leverages video to predict the long term behavior of specific objects of interest.

To interact with the environment, first we need to know where we are relative to it. Observability of the underlying dynamical model is a necessary condition for ANY algorithm to work, in the sense of yielding a unique point estimate. Our contribution is to show that all existing analysis of observability was flawed, and propose new analysis that shows that, contrary to popular belief, pose is not observable from visual and inertial sensors. However, we show that the ambiguous set is bounded, and compute it analytically. Once we know where we are, We need to know what is around us. This is a problem called “mapping”. Building geometric maps (point clouds) well explored problem. However, to interact intelligently need more than point cloud, we need some understanding of topology. How is the world around us divided into “objects”? Chapter 4 talks about a way of organizing points into surfaces and then connected components of surfaces, that can be considered “objects” for the purpose of interaction, from video. Once we know where we are and have a model of the (Visible) environment, with respect to which we know the location of an object of interest (point A), we need to know how to get to point A, which may not be visible. This requires exploration. Chapter 1 deals with this problem. The contribution is an efficient algorithm with provable bounds on the exploration time and amenable to be extended to non-compact domains (relevant in vision because one can see to infinity). To explore the boundaries of this problem set, we also ask whether a representation is needed at all, at least for simple problems like going to point To this end, we explore the possibility of directly encoding/representing/optimizing the map from sensory data to control action, designed so as to achieve the goal (of getting to point A).

Video analysis has evolved into a fundamental research problem following an unprecedented growth of video content consumption in the last few years. Its applications span a wide range of tasks including video editing, indexing, classification, surveillance and autonomous driving. While for these tasks, representations that capture the main properties of the scene have been investigated, this has still not been done for the purpose of video compression. Current video compression systems largely ignore the underlying scene and instead only model its projection on the video frames. The goal of this thesis is to show that by modeling the scene further improvements can be achieved in compressing video data. In the first part of the thesis, it is shown how a scene can be decomposed into two types of regions called ``Visual Structures" and ``Visual Textures". These regions are defined and algorithms that allow their inference are proposed. Based on such partition, a video compression system is designed that achieves a significant improvement over current state-of-the-art methods. In addition, the definition of textures is expanded to model transformations of the domain and range of the image and applications such as texture compression and segmentation are discussed. In the second part, extensions of the video compression system are proposed that take advantage of further modeling of the scene, such homogeneity, occlusions and tight boundaries of regions. Finally, further applications of scene modeling are shown in various problems such as video hole-filling and independent motion detection.

This thesis describes a research program towards enabling mobile visual agents to successfully solve visual decision problems. We identify perception, information acquisition, and prediction as the three areas to focus on, and develop methods to facilitate these stages, which are then applied to a sample selection of visual tasks. We first focus on segmentation and categorization of objects in video, and propose techniques for doing so. We then describe approaches for information-acquisition, useful whenever the data collection process can be controlled, or when one can choose which data to process. Finally, we describe an approach that leverages video to predict the long term behavior of specific objects of interest.

Video analysis has evolved into a fundamental research problem following an unprecedented growth of video content consumption in the last few years. Its applications span a wide range of tasks including video editing, indexing, classification, surveillance and autonomous driving. While for these tasks, representations that capture the main properties of the scene have been investigated, this has still not been done for the purpose of video compression. Current video compression systems largely ignore the underlying scene and instead only model its projection on the video frames. The goal of this thesis is to show that by modeling the scene further improvements can be achieved in compressing video data. In the first part of the thesis, it is shown how a scene can be decomposed into two types of regions called ``Visual Structures" and ``Visual Textures". These regions are defined and algorithms that allow their inference are proposed. Based on such partition, a video compression system is designed that achieves a significant improvement over current state-of-the-art methods. In addition, the definition of textures is expanded to model transformations of the domain and range of the image and applications such as texture compression and segmentation are discussed. In the second part, extensions of the video compression system are proposed that take advantage of further modeling of the scene, such homogeneity, occlusions and tight boundaries of regions. Finally, further applications of scene modeling are shown in various problems such as video hole-filling and independent motion detection.

To interact with the environment, first we need to know where we are relative to it. Observability of the underlying dynamical model is a necessary condition for ANY algorithm to work, in the sense of yielding a unique point estimate. Our contribution is to show that all existing analysis of observability was flawed, and propose new analysis that shows that, contrary to popular belief, pose is not observable from visual and inertial sensors. However, we show that the ambiguous set is bounded, and compute it analytically. Once we know where we are, We need to know what is around us. This is a problem called “mapping”. Building geometric maps (point clouds) well explored problem. However, to interact intelligently need more than point cloud, we need some understanding of topology. How is the world around us divided into “objects”? Chapter 4 talks about a way of organizing points into surfaces and then connected components of surfaces, that can be considered “objects” for the purpose of interaction, from video. Once we know where we are and have a model of the (Visible) environment, with respect to which we know the location of an object of interest (point A), we need to know how to get to point A, which may not be visible. This requires exploration. Chapter 1 deals with this problem. The contribution is an efficient algorithm with provable bounds on the exploration time and amenable to be extended to non-compact domains (relevant in vision because one can see to infinity). To explore the boundaries of this problem set, we also ask whether a representation is needed at all, at least for simple problems like going to point To this end, we explore the possibility of directly encoding/representing/optimizing the map from sensory data to control action, designed so as to achieve the goal (of getting to point A).

In a first chapter we describe a method to model perspective distortion as a one- parameter family of warping functions. This can be used to mitigate its effects on visual recognition, or interactively manipulate the perceived personality. The warps are learned from a novel face dataset and, by comparing orbits spanned by images instead of images themselves, we improve face recognition when small focal lengths are used. Additional applications are presented to image editing, videoconference, and multi-view validation of recognition systems.A second chapter is devoted to a new versatile and modular open-source cross- platform online object tracking library, designed to be easily usable by the vision community. Object tracking plays a central part in a number of vision problems, and there is no, to date, a ready-to-use and extensible tracking library at the object level.

In a first chapter we describe a method to model perspective distortion as a one- parameter family of warping functions. This can be used to mitigate its effects on visual recognition, or interactively manipulate the perceived personality. The warps are learned from a novel face dataset and, by comparing orbits spanned by images instead of images themselves, we improve face recognition when small focal lengths are used. Additional applications are presented to image editing, videoconference, and multi-view validation of recognition systems.A second chapter is devoted to a new versatile and modular open-source cross- platform online object tracking library, designed to be easily usable by the vision community. Object tracking plays a central part in a number of vision problems, and there is no, to date, a ready-to-use and extensible tracking library at the object level.

The modern world is predominantly visual. While most people live and function with vision, many people live their lives with limited or no vision at all. The system presented in this dissertation aims to use computer vision techniques to improve the lives of individuals having severe visual impairments or blindness. By applying real-time image recognition in an accessible manner the system facilitates increased personal independence. The system presented here substantially advances the state of the art both in terms of the speed, accuracy, and robustness to the real-world variability. The real-time on-device recognition speed enables more than just object identification; it permits detection -- finding the desired item among others. The high accuracy and robustness expands the ability for people to participate in commerce and expand roles for economic participation in multiple cultures. We present an extensive evaluation demonstration and a real-world public release of the system.

Possible object location generation is an important pre-process for most object detection algorithms. In this thesis, we design a window sampling algorithm to address possible object location generation problem in two steps. First, we use a two-phase feature space partition method to achieve local descriptor classification and find interest points on image which have high probability to be on object of interest. Then we introduced a way to learn the relationship between object bounding window and bag-of-words representation of local image region, with which we can sample windows that are highly possible to contain an object. We implement the algorithm in MATLAB and test it on Graz-02 dataset, which has three object categories: car, bike, human. The algorithm achieves state-of-the-art performance according to coverage, window quality, number of windows and running time. The MATLAB scripts are merged into one file called ``WindowMaskCode.pdf'', which can be found in supplementary files.

The modern world is predominantly visual. While most people live and function with vision, many people live their lives with limited or no vision at all. The system presented in this dissertation aims to use computer vision techniques to improve the lives of individuals having severe visual impairments or blindness. By applying real-time image recognition in an accessible manner the system facilitates increased personal independence. The system presented here substantially advances the state of the art both in terms of the speed, accuracy, and robustness to the real-world variability. The real-time on-device recognition speed enables more than just object identification; it permits detection -- finding the desired item among others. The high accuracy and robustness expands the ability for people to participate in commerce and expand roles for economic participation in multiple cultures. We present an extensive evaluation demonstration and a real-world public release of the system.