Search

We investigate some problems commonly found when dealing with stereo images. Working within the context of visual effects for films, we explore software solutions to issues arising with stereo images captured on-set. These images originate from a wide variety of hardware which may or may not provide additional data support for post-production needs. Generic software solutions are thus greatly to be preferred. This dissertation documents contributions in the following three areas. Each project was undertaken at Double Negative and investigated with the aim of improving the post-production pipeline for 3D films. Colour matching is the process whereby the colours of one view from a stereo pair are matched with those of the other view. This process is necessary due to the fact that slight differences in hardware and viewing angle can result in some surprisingly large colour discrepancies. Chapter 3 presents a novel approach to colour matching between stereo pairs of images, with a new tool for visual effects artists given in section 6.2.Vertical alignment of stereo images is key to providing a comfortable experience for the viewer, yet we are rarely presented with perfectly aligned footage from the outset. In chapter 4 we discuss the importance of correcting misalignments for both the final audience and the artists working on these images. We provide a tool for correcting misalignments in section 6.3.Disparity maps are used in many areas of post-production, and so in chapter 5 we investigate ways in which disparity map generation can be improved for the benefit of many existing tools at Double Negative. In addition, we povide an extensive exploration of the requirements of 3D films in order to make them presentable in the cinema. Through these projects, we have provided improvements to the stereo workflow and shown that academic research is a necessary component of developing tools for the visual effects pipeline. We have provided new algorithms to improve the 3D experience for moviegoers, as well as artists, and conclude by discussing the future work that will provide further gains in the field.

In this research work, we develop an obstacle detection and emergency exit sign recognition system on a mobile phone by extending the feature from accelerated segment test detector with Harris corner filter. The first step often required for many vision based applications is the detection of objects of interest in an image. Hence, in this research work, we introduce emergency exit sign detection method using colour histogram. The hue and saturation component of an HSV colour model are processed into features to build a 2D colour histogram. We backproject a 2D colour histogram to detect emergency exit sign from a captured image as the first task required before performing emergency exit sign recognition. The result of classification shows that the 2D histogram is fast and can discriminate between objects and background with accuracy. One of the challenges confronting object recognition methods is the type of image feature to compute. In this work therefore, we present two feature detectors and descriptor methods based on the feature from accelerated segment test detector with Harris corner filter. The first method is called Upright FAST-Harris and binary detector (U-FaHB), while the second method Scale Interpolated FAST-Harris and Binary (SIFaHB). In both methods, feature points are extracted using the accelerated segment test detectors and Harris filter to return the strongest corner points as features. However, in the case of SIFaHB, the extraction of feature points is done across the image plane and along the scale-space. The modular design of these detectors allows for the integration of descriptors of any kind. Therefore, we combine these detectors with binary test descriptor like BRIEF to compute feature regions. These detectors and the combined descriptor are evaluated using different images observed under various geometric and photometric transformations and the performance is compared with other detectors and descriptors. The results obtained show that our proposed feature detector and descriptor method is fast and performs better compared with other methods like SIFT, SURF, ORB, BRISK, CenSurE. Based on the potential of U-FaHB detector and descriptor, we extended it for use in optical flow computation, which we termed the Nearest-flow method. This method has the potential of computing flow vectors for use in obstacle detection. Just like any other new methods, we evaluated the Nearest flow method using real and synthetic image sequences. We compare the performance of the Nearest-flow with other methods like the Lucas and Kanade, Farneback and SIFT-flow. The results obtained show that our Nearest-flow method is faster to compute and performs better on real scene images compared with the other methods. In the final part of this research, we demonstrate the application potential of our proposed methods by developing an obstacle detection and exit sign recognition system on a camera phone and the result obtained shows that the methods have the potential to solve this vision based object detection and recognition problem.

It is well understood that multiple video cameras and computer vision (CV) technology can be used in sport for match officiating, statistics and player performance analysis. A review of the literature reveals a number of existing solutions, both commercial and theoretical, within this domain. However, these solutions are expensive and often complex in their installation. The hypothesis for this research states that by considering only changes in ball motion, automatic event classification is achievable with low-cost monocular video recording devices, without the need for 3-dimensional (3D) positional ball data and representation. The focus of this research is a rigorous empirical study of low cost single consumer-grade video camera solutions applied to table tennis, confirming that monocular CV based detected ball location data contains sufficient information to enable key match-play events to be recognised and measured. In total a library of 276 event-based video sequences, using a range of recording hardware, were produced for this research. The research has four key considerations: i) an investigation into an effective recording environment with minimum configuration and calibration, ii) the selection and optimisation of a CV algorithm to detect the ball from the resulting single source video data, iii) validation of the accuracy of the 2-dimensional (2D) CV data for motion change detection, and iv) the data requirements and processing techniques necessary to automatically detect changes in ball motion and match those to match-play events. Throughout the thesis, table tennis has been chosen as the example sport for observational and experimental analysis since it offers a number of specific CV challenges due to the relatively high ball speed (in excess of 100kph) and small ball size (40mm in diameter). Furthermore, the inherent rules of table tennis show potential for a monocular based event classification vision system. As the initial stage, a proposed optimum location and configuration of the single camera is defined. Next, the selection of a CV algorithm is critical in obtaining usable ball motion data. It is shown in this research that segmentation processes vary in their ball detection capabilities and location out-puts, which ultimately affects the ability of automated event detection and decision making solutions. Therefore, a comparison of CV algorithms is necessary to establish confidence in the accuracy of the derived location of the ball. As part of the research, a CV software environment has been developed to allow robust, repeatable and direct comparisons between different CV algorithms. An event based method of evaluating the success of a CV algorithm is proposed. Comparison of CV algorithms is made against the novel Efficacy Metric Set (EMS), producing a measurable Relative Efficacy Index (REI). Within the context of this low cost, single camera ball trajectory and event investigation, experimental results provided show that the Horn-Schunck Optical Flow algorithm, with a REI of 163.5 is the most successful method when compared to a discrete selection of CV detection and extraction techniques gathered from the literature review. Furthermore, evidence based data from the REI also suggests switching to the Canny edge detector (a REI of 186.4) for segmentation of the ball when in close proximity to the net. In addition to and in support of the data generated from the CV software environment, a novel method is presented for producing simultaneous data from 3D marker based recordings, reduced to 2D and compared directly to the CV output to establish comparative time-resolved data for the ball location. It is proposed here that a continuous scale factor, based on the known dimensions of the ball, is incorporated at every frame. Using this method, comparison results show a mean accuracy of 3.01mm when applied to a selection of nineteen video sequences and events. This tolerance is within 10% of the diameter of the ball and accountable by the limits of image resolution. Further experimental results demonstrate the ability to identify a number of match-play events from a monocular image sequence using a combination of the suggested optimum algorithm and ball motion analysis methods. The results show a promising application of 2D based CV processing to match-play event classification with an overall success rate of 95.9%. The majority of failures occur when the ball, during returns and services, is partially occluded by either the player or racket, due to the inherent problem of using a monocular recording device. Finally, the thesis proposes further research and extensions for developing and implementing monocular based CV processing of motion based event analysis and classification in a wider range of applications.

Understanding the motion of a camera from only the image(s) it captures is a di cult problem. At best we might hope to estimate the relative motion between camera and scene if we assume a static subject, but once we start considering scenes with dynamic content it becomes di cult to di↵erentiate between motion due to the observer or motion due to scene movement. In this thesis we show how the invaluable cues provided by inertial sensor data can be used to simplify motion analysis and relax requirements for several computer vision problems. This work was funded by the University of Bath.

This thesis presents a framework for a hybrid model-free marker-less inertial-visual camera pose tracking with an integrated sensor fusion mechanism. The proposed solution addresses the fundamental problem of pose recovery in computer vision and robotics and provides an improved solution for wide-area pose tracking that can be used on mobile platforms and in real-time applications. In order to arrive at a suitable pose tracking algorithm, an in-depth investigation was conducted into current methods and sensors used for pose tracking. Preliminary experiments were then carried out on hybrid GPS-Visual as well as wireless micro-location tracking in order to evaluate their suitability for camera tracking in wide-area or GPS-denied environments. As a result of this investigation a combination of an inertial measurement unit and a camera was chosen as the primary sensory inputs for a hybrid camera tracking system. After following a thorough modelling and mathematical formulation process, a novel and improved hybrid tracking framework was designed, developed and evaluated. The resulting system incorporates an inertial system, a vision-based system and a recursive particle filtering-based stochastic data fusion and state estimation algorithm. The core of the algorithm is a state-space model for motion kinematics which, combined with the principles of multi-view camera geometry and the properties of optical flow and focus of expansion, form the main components of the proposed framework. The proposed solution incorporates a monitoring system, which decides on the best method of tracking at any given time based on the reliability of the fresh vision data provided by the vision-based system, and automatically switches between visual and inertial tracking as and when necessary. The system also includes a novel and effective self-adjusting mechanism, which detects when the newly captured sensory data can be reliably used to correct the past pose estimates. The corrected state is then propagated through to the current time in order to prevent sudden pose estimation errors manifesting as a permanent drift in the tracking output. Following the design stage, the complete system was fully developed and then evaluated using both synthetic and real data. The outcome shows an improved performance compared to existing techniques, such as PTAM and SLAM. The low computational cost of the algorithm enables its application on mobile devices, while the integrated self-monitoring, self-adjusting mechanisms allow for its potential use in wide-area tracking applications.

Image processing algorithms have developed rapidly in recent years. Imaging functions are becoming more common in electronic devices, demanding better image quality, and more robust image capture in challenging conditions. Increasingly more complicated algorithms are being developed in order to achieve better signal to noise characteristics, more accurate colours, and wider dynamic range, in order to approach the human visual system performance levels.

Estimating the dense image motion or optical flow on a real-world nonrigid surface is a fundamental research issue in computer vision, and is applicable to a wide range of fields, including medical imaging, computer animation and robotics. However, nonrigid surface tracking is a difficult challenge because complex nonrigid deformation, accompanied by image blur and natural noise, may lead to severe intensity changes to pixels through an image sequence. This violates the basic intensity constancy assumption of most visual tracking methods. In this thesis, we show that local geometric constraints and long term feature matching techniques can improve local motion preservation, and reduce error accumulation in optical flow estimation. We also demonstrate that combining RGB data with additional information from other sensing channels, can improve tracking performance in blurry scenes as well as allow us to create nonrigid ground truth from real world scenes. First, we introduce a local motion constraint based on a laplacian mesh representation of nonrigid surfaces. This additional constraint term encourages local smoothness whilst simultaneously preserving nonrigid deformation. The results show that our method outperforms most global constraint based models on several popular benchmarks. Second, we observe that the inter-frame blur in general video sequences is near linear, and can be roughly represented by 3D camera motion. To recover dense correspondences from a blurred scene, we therefore design a mechanical device to track camera motion and formulate this as a directional constraint into the optical flow framework. This improves optical flow in blurred scenes. Third, inspired by recent developments in long term feature matching, we introduce an optimisation framework for dense long term tracking -- applicable to any existing optical flow method -- using anchor patches. Finally, we observe that traditional nonrigid surface analysis suffers from a lack of suitable ground truth datasets given real-world noise and long image sequences. To address this, we construct a new ground truth by simultaneously capturing both normal RGB and near-infrared images. The latter spectrum contains dense markers, visible only in the infrared, and represents ground truth positions. Our benchmark contains many real-world scenes and properties absent in existing ground truth datasets.

This thesis considers three inverse problems originating in mathematical imaging, remote sensing and simulation. First, it addresses a challenging image processing task: recovering images corrupted by multiplicative noise. Motivated by the success of multiscale hierarchical decomposition methods (MHDM) in image processing, a variety of both classical and new multiplicative noise removing models are adapted to the MHDM form. On the basis of previous work, tight and refined extensions of the multiplicative MHDM process are proposed. Existence and uniqueness of solutions for the proposed models is discussed, in addition to convergence properties. Moreover, the work introduces a discrepancy principle stopping criterion which prevents recovering excess noise in the multiscale reconstruction. The validity of all the proposed models is qualitatively and quantitatively evaluated through comprehensive numerical experiments and comparisons across several images degraded by multiplicative noise. By construction, these multiplicative multiscale hierarchical decomposition methods have the added benefit of recovering many scales of an image, which can provide features of interest beyond image denoising. Second, this work considers an applied mathematical imaging problem in remote sensing. Accurate estimation of atmospheric wind velocity plays an important role in weather forecasting, flight safety assessment and cyclone tracking. Atmospheric data captured by infrared and microwave satellite instruments provide global coverage for weather analysis. Extracting wind velocity fields from such data has traditionally been done through feature tracking, correlation/matching or optical flow means from computer vision. However, these recover either sparse velocity estimates, oversmooth details or are designed for quasi-rigid body motions which over-penalize vorticity and divergence within the often turbulent weather systems. This thesis proposes a texture based optical flow procedure tailored for water vapor data. The method implements an $L1$ data term and total variation regularizer and employs a structure-texture image decomposition to identify key features which improve recoveries and help preserve the salient vorticity and divergence structures. Based on the success of texture-features at improving the flow estimation, this procedure is extended to a multi-fidelity scheme which incorporates additional image features into the optical flow scheme. Both flow estimation methods are tested on simulated over-ocean mesoscale convective systems and convective and extratropical cyclone datasets, each of which have accompanying ground truth wind velocities allowing quantitative comparisons of each method's performance against existing optical flow methods. Third, the thesis considers a data-driven reduced-order modeling problem. It is well known that full-resolution simulations of turbulence is computationally expensive (and often intractable), motivating the development of reduced-order models for practical engineering and science applications. However, developing accurate reduced models for turbulent flow is impeded by the loss of unresolved scales which critically influence the full state. The Mori-Zwanzig formalism is a mathematically exact strategy for performing reduced-order modeling in which reduced-order variables (observables) are evolved by memory kernels and an orthogonal dynamics term. Recently, frameworks for extracting Mori-Zwanzig kernels from simulated data have shown promise for reduced turbulence models. However, these same works highlight the importance of choosing an appropriate reduced-order observables set, and how this choice can substantially affect the model's overall success. This thesis proposes a joint-learning process to discover an improved set of observables in tandem with extracting the Mori-Zwanzig memory kernels. The process is tested on simulated two-dimensional turbulence data, and results show that the discovered set of observables enhances the Mori-Zwanzig based turbulence model's capabilities at predicting turbulent structures and statistics within the resolved space, both in short and long trajectories.

The thesis investigates obstacle detection and path planning methods appropriate for use in an unmanned helicopter, which are essential to establishing autonomous control of the vehicle. The research described here is complementary to a feasibility study by EA Technology Ltd to consider the use of robotic vehicles in the inspection of overhead electricity power lines. The context of this project is the proposal that the inspection be performed by a small, remotely-piloted helicopter fitted with a video camera, returning video imagery to the ground-based observer. A major obstacle to this concept is that, under current flight regulations, the helicopter would not be allowed to fly outside the visual range of the ground station. This work investigates whether autonomous obstacle detection based on machine vision coupled with 3D path planning has the potential to remove t his constraint. The thesis first discusses the requirements and current methods of overhead line inspection. A review of relevant machine vision and path planning methods is given and a method for placing these into a hierarchical control system architecture is described. A method for fast 3D path planning, based on the distance transform, is introduced and experimental results are presented to show its effectiveness. Implementation of the control system, integrating the vision system and path planner, is done by means of a scale model laboratory test rig. The construction of this test rig is described and results are presented to show obstacle avoidance in action. The thesis ends with an assessment of how far the research has advanced the prospect of autonomous aerial power line inspection.

Future frame prediction is a difficult but useful problem to solve in deep learning. The technology can be used to predict future occurrences in a video, anticipate anomalies, and aid autonomous devices in smart decision making. Although there is potential with frame prediction technology, there is still progress that needs to be made with it. As the predicted frame becomes farther away from the last input frame, the image becomes blurry and distorted. This indicates that the model is more uncertain about the motion occurring in the image frame. To reduce model uncertainty shown in predictions, optical flow information from each video was extracted and combined with the video frames. An optical flow-based approach is uncommon in frame prediction and has not been implemented with a fully Convolutional Neural Network (CNN) based architecture. In this work, the change in image quality evaluation metrics and overall image quality is analyzed across 4 different datasets between a state-of-the-art frame prediction model and a modified model that combines optical flow information. The results demonstrate that adding optical flow information improves the model Mean Squared Error (MSE) by 4.11% and its Structural Similarity Index Metric (SSIM) by 0.41% for the Moving MNIST dataset. Optical flow improved the SSIM value of Taxi BJ, KTH, and KITTI by 0.02%, 0.011%, and 1.297% respectively. While there was a consistent improvement in performance, the models still need more improvement in terms of the quality of images predicted in the distant future.

Simultaneous Localization and Mapping (SLAM) is the process of using distance measurements to points in the surrounding environment to build a digital map and perform localization. It has been observed that featureless environments like tunnels or straight hallways will cause positioning faults in SLAM. This research investigates the fault diagnosis and accommodation problem for a laser-rangefinder-based SLAM systems on a quadrotor. A potential solution of using optical flow as velocity estimate and an extended Kalman filter (EKF) to perform position estimation is proposed. A fault diagnosis method for detecting faults in positional SLAM data or optical flow velocity data is developed by using two parallel EKFs. When a fault in the SLAM position or optical flow velocity is detected, the EKF adapts to provide a robust position estimate to ensure the safety of the flight control system.

The optical flow task is an active research domain in the machine learning community with numerous downstream applications such as autonomous driving and action recognition. Convolutional neural networks have often been the basis of design for these models. However, given the recent popularity of self-attention architectures, many optical flow models now utilize a vision transformer backbone. Despite the differing backbone architectures used, consistencies in model design, especially regarding auxiliary operations, have been observed. Perhaps most apparent is the calculation and use of the cost volume. While prior works have well documented the effects of the cost volume on models with a convolutional neural network backbone, similar research does not exist for optical flow models based on other architectures, such as the vision transformer. Naturally, a research question arises: what are the effects of utilizing a cost volume in vision transformer-based optical flow models? In this thesis, a series of experiments examine the impact of the cost volume on training time, model accuracy, average inference time, and model size. The observed results show that cost volume use increases the model size and training time while improving model accuracy and reducing average inference time. These results differ from those regarding the effects of the cost volume on convolutional neural network-based models. With the results presented, researchers can now adequately consider the potential benefits and drawbacks of cost volume use in vision transformer-based optical flow models.

Although virtual reality has been gaining in popularity, users continue to report discomfort during and after the use of VR applications, and many experience symptoms associated with cybersickness. To mitigate this problem, dynamic field-of-view (FOV) restriction is a common technique that has been widely implemented in commercial VR games. FOV restriction artificially reduces the field of view during movement to limit optical flows and reduce discomfort caused by the mismatch between virtual motion and physical motion. FOV restriction has been shown in numerous studies to improve comfort and enhance user experience in virtual reality. The standard dynamic FOV restriction is created by adding a symmetrical black opaque mask at the periphery of the user’s filed-of-view and its size changes only with the user's virtual velocity. It does not take into account any differences in users, virtual environments, or other usage conditions. This simplistic implementation leads to some limitations. The first limitation is that the FOV restriction reduces users' visibility of the virtual environment and can negatively impact their subjective experience. The second limitation is that the unblocked imagery when applying restrictor is usually in the center of the field of view, which is incompatible with the users' eye movements during locomotion. The third limitation is that the classical restrictor is scaled by the velocity and angular velocity of users' virtual movements. This design assumes that users feel the same cybersickness when they experience the same velocity, which is unrealistic. Beyond these limitations, there is a lack of scientific understanding of how to effectively apply FOV restrictions for different types of virtual environments and virtual motions. This thesis presents four major contributions to the existing dynamic FOV restriction research. First, I present a novel technique known as passthrough FOV restriction, which combines the dynamic field of view modification with rest frames generated from 3D scans of the physical environment. The informal testing suggests that this approach is a promising method for reducing motion sickness and improving user safety at the same time. Secondly, I present a novel asymmetric field-of-view restrictor known as the ground-visible restrictor, which maintains the visibility of the ground plane during movement. User studies showed that ground-visible FOV restriction offers benefits for user comfort, postural stability, and a subjective sense of presence. Thirdly, I provide another variant of FOV restriction, referred to as the side restrictor, which expands side visibility and maintains restriction during rotation. A user study evaluated the new technology and demonstrated its benefits in reducing cybersickness and discomfort and improving visibility. Finally, I present an adaptive restrictor that uses the optical flow amount to determine the position and size of the restriction. A mixed design study investigated its performance and confirmed its superiority over traditional restrictors in providing a better subjective user experience.

Over the years, fully autonomous vehicles(AV) have made significant progress towards being employed in real-life scenarios, although they still lack enough sophistication to drive through a completely unknown environment. Real-life scenarios such as during search and rescue operations, infrastructure inspection, or even indoor navigation demands AVs to work in an unfamiliar GPS-denied environment. This kind of situation is usually dealt as SLAM problems and solved using sensors like LiDAR, Stereo, and Monocular cameras. Even though approaches using LiDAR and Stereo cameras have shown some promising results, they require significant amounts of processing power and occupy considerable space, two factors that are inherently limited with an in-vehicle computer system. An alternative is to use a monocular camera that is both easy to install and light on processing. However, they suffer from an absence of depth data to perceive the 3D world around it to successfully perform SLAM. This research addresses the problem by proposing ``MonoDepth-vSLAM" - a novel robust visual SLAM algorithm using a single monocular camera. The algorithm uses Extended Kalman Filter with an optical flow-based model for real-time pose estimation and feature tracking, assisted by monocular depth estimation obtained from self-supervised neural networks. The classical EKF approach provides a lightweight method for tracking vehicle states with uncertainty bounds, whereas the depth estimation network mitigates the absence of depth information required to perform monocular SLAM successfully. The proposed algorithm is designed to work in real-time and has been tested and analyzed in different scenarios.

A collection of images of a scene captured from different perspectives inform us about the scene's composition. Visualizing different perspectives of a scene in intuitive ways using simple techniques, like light-field visualization and panorama stitching, allow us to maximize the perception and understanding of a scene. While visualizing a collection of images in intuitive ways allows humans to consume the image content in a much richer way, it can also provide better ways for computer algorithms to consume the novel visualizations of the captured images and reason about the 3D information in the scene. In this thesis, we develop novel techniques to explore these two threads. To generate image-based visualizations, we use plane-induced homography transformations to warp and compare images.Plane-induced homography transformations allow us to warp and align multiple images with different perspectives of a scene to a common 3D reference plane. We use this simple image warping operation, combined with the knowledge of the camera motion, to analyze the characteristics of flow between the warped images for the objects that lie on and off the 3D reference plane. We leverage the observed characteristics of the flow between homography-warped images to develop a computer graphics application for 3D scene visualization for human consumption. We also use it to develop a new learning-based framework to solve some core 3D computer vision tasks. More specifically, we investigate the following two application threads:1) We propose a new framework for visualizing multiple perspectives of a scene and even combine them into a single multi-perspective image by rendering the 3D scene to a multi-perspective camera projection mapping. We showcase this via a computational photography application. By taking as input a collection of images captured by a dolly-in camera motion, we allow the photographers to change the relative size of imaged objects, change the sense of depth of the scene, and control the foreground perspective distortion in the final image composition. This allows photographers intuitive control for post-capture image composition. 2) We propose a new framework that uses multiple visualizations for a pair of images to pose simple questions about the 3D scene information to a neural-network and train it to provide binary answers. We use this framework and show that the core 3D computer vision problems like depth estimation for static scenes, and time-to-contact and optical-flow estimation for dynamic scenes can be solved via binary comparisons. Each binary comparison gives meaningful information in just a few milliseconds: Is an object in-front or beyond a 3D plane in the scene? Is an object's optical flow greater or smaller than a specified threshold? Will an object, static or dynamic, hit the camera plane within a specified time-to-contact? We also show that multiple binary comparisons could be combined together to get quantized or even continuous estimation when computational budget allows for it. Our framework achieves competitive performance with state-of-the-art results while providing a principled way to trade accuracy for latency at inference time, thus providing a useful tool for many robotics applications.

This thesis deals with the problem of online motion planning for a vehicle when the available environmental information is limited. It is expected that the car moves within a lane using only the ego information (global x; y; z position and roll, pitch and yaw angles) and the information from a camera mounted in the windshield without previous knowledge of the road (like a map). The images obtained from the camera are used to obtain the optical flow, i.e., the measure of the apparent motion of the environment ahead of the moving vehicle.The continuously changing optical flow allows recognizing some features on the road, which are then used for a road artificial potential field computation. That field is used to generate an online reference trajectory (desired orientation) for the vehicle.Then, a gradient tracking control based on sliding mode is used to make the vehicle move along the prescribed trajectory determined by the gradient lines. A regular sliding mode controller is used for the longitudinal tracking of speed. Carla simulator is used for deployment and testing, and some different weathers are tested to evaluate the performance of the proposed approach.

Optical flow (OF) which refers to the task of determining the apparent motion of objects in the scene has been a topic of core interest in commuter vision for the past three decades. Optical flow methods of conventional camera struggle in the presence of large motion and occlusion due to slow frame rates. Optical flow of dynamic vision sensor (DVS) has gained attention recently as a way to overcome these shortcomings. DVS known also as event detection camera emerged recently as an alternative to a conventional camera by replacing a fixed analog-to-digital (A/D) converter with a floating asynchronous circuit. Rather than reporting a pixel intensity at a fixed time interval, the event detection cameras report only the significant changes (i.e. above threshold) to the pixel intensity (the “events”) and the time that such event occurs. Such circuit significantly reduces the communication bandwidth of the camera, enabling the operation at equivalent of roughly 80,000 frames per second. In addition, the floating A/D converter may adapt to extremely high dynamic range, making it suitable for applications in automotives and scientific instruments. However, the sparsity of the output data renders existing image processing and computer vision methods useless. For example, the “brightness constancy constraint” that is at the heart of optical flow does not apply to the edge-like features that event detection cameras output, and the very notion of “frames” is absent in the asynchronous outputs. In this work, we consider a new sensor called DAViS that combines the conventional active pixel sensor (APS) and DVS circuitries, yielding a conventional intensity image frames as well as the events. We propose three novel optical flow methods: First, We propose a novel optical flow method designed specifically for a DAViS camera that leverages the high spatial fidelity of intensity image frames and the high temporal resolution of events generated by DVS. Hence, the proposed DAViS-OF method yields reliable motion vector estimates while overcoming the fast motion and occlusion problems. Secondly, we develop a novel DVS optical flow method using the 2D distance transform---computed from the detected events---as a proxy for object texture. Treating multiple 2D distance transforms collectively as a ``distance surface'' improves optical flow significantly over working directly with the sparse events generated by the DVS camera. Finally, we introduce a new DVS-based method that extend the notion of 2D distance transform into 3D distance transform by incorporating the temporal information, resulting in reliable optical flow estimate. Real sensor experiments verify the accuracy and robustness of proposed methods to reliably recover the true two dimensional pixel motion, not limited to the ``normal flow.''

Nowcasting precipitation, that is, accurately predicting its location and intensity minutes to a few hours ahead, is a difficult task. Extreme rainfall events can be a threat to the population and challenge the limits of weather monitoring and prediction systems. For practical use, the forecasting techniques at our disposal range from physically-based numerical models to heuristic extrapolation procedures based on weather radars. Radar-based nowcasting models rely on the high spatial and temporal resolution of radar measurements and thus benefit from the best possible initial conditions, but the assumption of persistence leads to a rapid decay of predictive skill with increasing lead time and decreasing spatial scale. Numerical models provide physically consistent precipitation forecasts, but their practical relevance for nowcasting can be undermined by uncertain initial conditions, spinup issues, model approximations, or simply by their computational limits. Analyses presented in this study show that, on average, radar-based nowcasting outperforms numerical simulations during the first three hours and that it is particularly useful for forecasting precipitation patterns with a horizontal dimension below 60 kilometers, as state-of-the-art numerical models cannot provide useful skill at those small spatial scales. An interesting finding is that the predictive uncertainty of numerical predictions relatively to radar nowcasting improves during warm convective days, which is explained by the combined effect of shorter precipitation lifetimes and more effective model assimilation of locally triggered air mass convection. After a lead time of 4.5 hours, we observed that precipitation on all scales below 150 kilometers is poorly predictable by all forecasting means. Such serious limits to predictability determine the need to represent forecast uncertainty as accurately as possible. In radar nowcasting, ensemble methods use stochastic simulations to perturb a deterministic extrapolation and thus quantify forecast errors. We found that a more precise representation of the statistical properties of the precipitation field through localization can have a positive impact on the realism of the simulations as well as in terms of probabilistic forecast skill. We showed that localized nowcasts perform better in terms of ensemble reliability and resolution, as well as conditional bias. However, we also found that a too strong a localization can lead to lower skill as it implicitly relies on a stricter assumption of persistence. The quantitative estimation of forecast uncertainty provided by ensembles was finally used to design a seamless blending procedure that integrates all available sources of predictive skill. Implemented using a recursive formulation of the Bayesian update equations, the blending scheme involves a prediction step through a stochastic radar extrapolation, while a subsequent correction step updates the extrapolation using information from the most recent numerical model run. It is found that such an approach is able to capture the flow dependence of both the numerical forecast and the radar nowcast ensemble spreads resulting in an adaptive blending scheme that depends on the relative uncertainty of the individual forecasts. Despite the non-Gaussian nature of rainfall data, we were able to produce blended precipitation forecasts that are at least as skillful as the radar-only or the numerical model-only forecasts at any lead time.

The power of deep neural networks comes mainly from huge labeled datasets. Even though it shines on many computer vision tasks, supervised learning bears little hope to hack into the core of intelligent visual systems. On the other side, unsupervised learning is believed to be the future of AI; however, its performance is always inferior compared to the supervised counterpart. The goal of our research is to develop unsupervised learning algorithms for computer vision tasks while matching or even outperforming the supervised ones. Our key is a representation that is as informative as the supervisory labels, which can be constructed from an unlimited amount of unlabeled data. In theory, this representation contains richer information than the processed supervisory signal. Moreover, we develop algorithms that can utilize existing labeled datasets to expedite the information extraction from the unlimited unlabeled data. Our research is lined up in an order similar to the visual development in early infancy, such that we can also investigate the interplay between different visual functionalities. The final goal is to develop a robotic visual system akin to a human's, that can automatically acquire semantics from concepts of objects fostered by basic perceptions of motion and depth with the minimum amount of human supervision.

In this dissertation, I present my work towards exploring temporal information for better video understanding. Specifically, I have worked on two problems: action recognition and semantic segmentation. For action recognition, I have proposed a framework, termed hidden two-stream networks, to learn an optimal motion representation that does not require the computation of optical flow. My framework alleviates several challenges faced in video classification, such as learning motion representations, real-time inference, multi-framerate handling, generalizability to unseen actions, etc. For semantic segmentation, I have introduced a general framework that uses video prediction models to synthesize new training samples. By scaling up the training dataset, my trained models are more accurate and robust than previous models even without modifications to the network architectures or objective functions. Along these lines of research, I have worked on several related problems. I performed the first investigation into depth for large-scale video action recognition where the depth cues are estimated from the videos themselves. I further improved my hidden two-stream networks for action recognition through several strategies, including a novel random temporal skipping data sampling method, an occlusion-aware motion estimation network and a global segment framework. For zero-shot action recognition, I proposed a pipeline using a large-scale training source to achieve a universal representation that can generalize to more realistic cross-dataset unseen action recognition scenarios. To learn better motion information in a video, I introduced several techniques to improve optical flow estimation, including guided learning, DenseNet upsampling and occlusion-aware estimation. I believe videos have much more potential to be mined, and temporal information is one of the most important cues for machines to perceive the visual world better.

Situational awareness and indoor positioning of firefighters are types of information of paramount importance to the success of search and rescue operations. GPS units are undependable for use in Indoor Positioning Systems due to their associated mar- gins of error in position and their reliance on satellite communication that can be interrupted inside large structures. There are few other techniques like dead reck- oning, Wifi and bluetooth based triangulation, Structure from Motion (SFM) based scene reconstruction for Indoor positioning system. However due to high temper- atures, the rapidly changing environment of fires, and low parallax in the thermal images, the above techniques are not suitable for relaying the necessary information in a fire fighting environment that is needed to increase situational awareness in real time. In fire fighting environments, thermal imaging cameras are used due to smoke and low visibility, hence obtaining relative orientation from camera information alone be- comes very difficult. We gained a deeper appreciation of the significance of maintain- ing a sense of relative orientation to a first responders ability to successfully navigate when we first visited first responders training facility for data collection. The follow- ing technique that is the content of this research is two fold; Firstly, we implement a novel, optical flow, gyroscope-based relative orientation estimation and secondly, we implement a velocity estimation technique using data from accelerometer fused with LIDAR. We also provide insight, discuss about the data we are going to use for our implementations. The Implementation helps first responders to go into unprepared, unknown envi- ronments and still maintain situational awareness like the orientation aid and position of the victim fire fighters. We also discuss and provide proof of concept implemen- tation to apply Deep Q learning algorithm on top of this to present efficient, safest path a firefighter can take in a first responding scenario.

Human visual perception is strongly dependent on recognition of object shape and motion. In particular, the algorithmic understanding of motion — observed scene motion, as well as the motion of the observer — is an important building block for most higher-level tasks in visual sensing and scene understanding from image sequences. The problem of motion estimation can be further divided into different sub-problems of varying difficulty, goals and underlying assumptions. The most general case of scene motion estimation is finding the observed motion at every image location from one frame of an image sequence to the next frame. The task of estimating this two-dimensional vector field is usually named Optical Flow. It does not require any assumptions about the observed scene nor the observer. It is, however, a high-dimensional and computationally demanding task. Furthermore, it does not take into account that observed natural scenes can often be segmented into groups of objects with similar and smooth motion. This observation can be exploited for object motion estimation by assuming that the spatial grouping and segmentation of image content at each time instance into objects is available through other means, for instance by using frame-wise object detections. The motion estimation task is then simplified to finding a coherent temporal grouping and motion estimation on object level. Finding and temporally grouping such sets of similar objects is often referred to as tracking-by-detection. If multiple synchronized image sequences, observing the same scene, are available, the temporal grouping can be extended to jointly find consistent groups over time and multiple views. Observed motion is the combination of motion induced by the observer (i.e. moving camera) and object motion. Often, the observed scene can be segmented into a non-moving background and multiple moving objects. In those cases separating the tasks of estimating the motion of the observer and object motion is possible. It is, however, an ill-defined problem to identify and separate a static background from moving objects, without knowledge or priors on the observer motion. Therefore, it is often easier to focus on recovering the observer motion first, by assuming that the motion of the static background dominates, before proceeding to object motion. If a precise model of the static 3D geometry of the observed scene is available, the observer motion can be recovered by registering the image sequence to this model. Another cue to help estimating the orientation of the observer and the geometry of the static environment in scenes depicting man-made structures are the convergence points of imaged scene-parallel structures, i.e. vanishing points. This thesis is a contribution to the estimation and understanding of scene and observer motion from image sequences in these fields. We will describe our proposed method to efficiently compute generic two-dimensional optical flow, as published in [Kroeger et al., 2016], our method for multi-target tracking-by-detection across multiple sequences in the presence of camera localization noise, as published in [Kroeger et al., 2014], our methods for vanishing point detection and tracking, as published in [Kroeger et al., 2015a] and [Kroeger et al., 2015b], and our method for video registration to given structure-from-motion 3D-models, as published in [Kroeger and Van Gool, 2014].

This thesis presents a deep neural network model that augments an existing semanticimage segmentation model with optical flow data to improve segmentation performanceon video sequences. Three network topologies combining optical flow datalayers with RGB data layers are compared. The best performing model, FlowSegA,achieves an average per-class accuracy of 72.696% on the SegNet test set. Thisis an improvement of 4.8 percentage-points versus SegNet, the RGB-only segmentationmodel on which FlowSeg-A is based. The main accuracy improvements comefrom the classes SignSymbol (15.4% improvement), Bicyclist(10.2%), and Pole(9.0%). These accuracy improvements are achieved with only 1,152 (0.005%) moreparameters, and FlowSeg-A achieves this performance using the same training set andtraining schedule as the SegNet algorithm.

Current preclinical screening methods are ineffective at detecting cardiotoxicity: 30% of drug attritions are attributed to drug-induced cardiotoxicity. With recent advancements in stem cell technologies, human pluripotent stem cells-derived cardiomyocytes (hPSC-CM) can now provide a physiologically relevant in vitro model of the myocardium. Availability of such cells has led to the emergence of various platforms that utilize them. However, current platforms still encounter challenges that deter their adoption for commercial use. One issue is hPSC-CMs exhibit fetal-like phenotypes; screening with them leads to unpredictable results that don’t accurately represent cardiotoxicity in adults. Another persistent challenge is the need to develop a simple and reliable method to measure key electrophysiological and contractile parameters. In addition, analytical approaches need to be created for accurate and automated detection of cardiotoxicity from platform readouts. In this thesis, strategies that collectively form a platform are described to address these issues. Alignment of hPSC-CMs has been shown to regulate sarcomere orientation, produce stronger contractile forces, and cause anisotropic action potential propagation. We demonstrate that biomimetic substrates with topographical alignment cues (uniaxial and multi-scale ‘wrinkles’) can be fabricated by using pre-stressed thermoplastic shrink film. These wrinkles recapitulate the anisotropic nature of the ECM of the native myocardium. When aligned on these substrates, hPSC-CMs exhibited a more sensitive response to cardioactive compounds than their unaligned counterparts. Using brightfield microscopy and optical flow, contractility of hPSC-CMs exposed to compounds can be monitored in a non-invasive and inexpensive manner. Furthermore this brightfield technique was readily applied to cardiac constructs of various tissue geometries, ranging from 2D monolayers to 3D cardiac organoids. For improved and automated analysis, we mated the brightfield technique with supervised machine learning. The machine learning provides a singular quantitative index that summarizes the impact of multiple parameters, and thus simplifies the assessment of drug effects on hPSC-CMs. Through the evaluation of several cardioactive drugs with dissimilar effects, this paired method was comparable – and even superior to – a fluorescence-based detection scheme common in commercially available systems. The machine learning was further leveraged for the analysis of a cardiac tissue strip platform. A model of drug classes was created and successfully predicted the mechanistic action of an unknown cardioactive compound.

Within the fields of visual effects and animation, humans have historically spent countless painstaking hours mastering the skill of drawing frame-by-frame animations. One such animation technique that has been widely used in the animation and visual effects industry is called "rotoscoping" and has allowed uniquely stylized animations to capture the motion of real life action sequences, however it is a very complex and time consuming process. Automating this arduous technique would free animators from performing frame by frame stylization and allow them to concentrate on their own artistic contributions. This thesis introduces a new artificial system based on an existing neural style transfer method which creates artistically stylized animations that simultaneously reproduce both the motion of the original videos that they are derived from and the unique style of a given artistic work. This system utilizes a convolutional neural network framework to extract a hierarchy of image features used for generating images that appear visually similar to a given artistic style while at the same time faithfully preserving temporal content. The use of optical flow allows the combination of style and content to be integrated directly with the apparent motion over frames of a video to produce smooth and visually appealing transitions. The implementation described in this thesis demonstrates how biologically-inspired systems such as convolutional neural networks are rapidly approaching human-level behavior in tasks that were once thought impossible for computers. Such a complex task elucidates the current and future technical and artistic capabilities of such biologically-inspired neural systems as their horizons expand exponentially. Further, this research provides unique insights into the way that humans perceive and utilize temporal information in everyday tasks. A secondary implementation that is explored in this thesis seeks to improve existing convolutional neural networks using a biological approach to the way these models adapt to their inputs. This implementation shows how these pattern recognition systems can be greatly improved by integrating recent neuroscience research into already biologically inspired systems. Such a novel hybrid activation function model replicates recent findings in the field of neuroscience and shows significant advantages over existing static activation functions.

The inclusion of precision dairy farming technologies in dairy operations is an area of increasing research and industry direction. Machine vision based systems are suitable for the dairy environment as they do not inhibit workflow, are capable of continuous operation, and can be fully automated. The research of this dissertation developed and tested 3 machine vision based precision dairy farming technologies tailored to the latest generation of RGB+D cameras. The first system focused on testing various imaging approaches for the potential use of machine vision for automated dairy cow feed intake monitoring. The second system focused on monitoring the gradual change in body condition score (BCS) for 116 cows over a nearly 7 month period. Several proposed automated BCS systems have been previously developed by researchers, but none have monitored the gradual change in BCS for a duration of this magnitude. These gradual changes infer a great deal of beneficial and immediate information on the health condition of every individual cow being monitored. The third system focused on automated dairy cow feature detection using Haar cascade classifiers to detect anatomical features. These features included the tailhead, hips, and rear regions of the cow body. The features chosen were done so in order to aid machine vision applications in determining if and where a cow is present in an image or video frame. Once the cow has been detected, it must then be automatically identified in order to keep the system fully automated, which was also studied in a machine vision based approach in this research as a complimentary aspect to incorporate along with cow detection. Such systems have the potential to catch poor health conditions developing early on, aid in balancing the diet of the individual cow, and help farm management to better facilitate resources, monetary and otherwise, in an appropriate and efficient manner. Several different applications of this research are also discussed along with future directions for research, including the potential for additional automated precision dairy farming technologies, integrating many of these technologies into a unified system, and the use of alternative, potentially more robust machine vision cameras.

This thesis focuses on the issue of real-time low speed flight of Micro-Air-Vehicles (MAVs) in cluttered environments. In order to achieve real-time autonomous operation, we have developed a navigation system for a Micro-Air-Vehicle using a RGB-D camera. Low-cost range sensors are perfect alternatives for expensive laser scanners in many application areas. For our experiments, we make use of a recent development in consumer-grade range sensing technology, the Microsoft Kinect sensor, consisting of an infrared laser emitter, an infrared camera and a RGB camera. Dominant plane estimation is a fundamental task for obstacle detection and autonomous navigation of MAVs. A modified region growing technique for plane detection is developed with an incremental approach, optimised plane and mean square error calculations, resulting in improvements to accuracy and efficiency. The Image Interpolation Algorithm (I2A) is used to calculate optical flow from the RGB-D camera intensity image sequences and corresponding depth information. A novel ego-motion recovery method is proposed by combining the range data with the optical flow field Image Jacobian. For our algorithm, assumptions about the moving object or dynamic scenes are not needed. An Extended Kalman Filter is used to fuse inertial data with the ego-motion parameters. By integrating the egomotion, estimation of the velocity and position of the quadrotor is obtained in three dimensional space. With lack of information about the surroundings, path planning in a cluttered environment is a very challenging task and fast re-planning is required when new obstacles are detected. The enhanced D* Lite algorithm is proposed for shortening the travel distance in cluttered environments and a novel dense scene flow estimation method is presented which can be used for detecting dynamic obstacles. An AscTec pelican quadrotor is used for real-time experiments. A Vicon Motion Tracking System provides the measurement parameters for ground truth in order to analyse the system error. Based on real-time experiments implemented in cluttered environments, the different parts of the research results are tested and validated.

The understanding of motion is an important problem in computer vision with applications including crowd-flow analysis, video surveillance, and estimating three-dimensional structure. A less-explored problem is the visual characterization and quantification of motion complexity. An important motion class that is prevalent in living beings is articulated motion (segments connected by joints). At present, no known standardized measure for quantifying the complexity of articulated motion exists. Such a measure could facilitate advanced motion analysis with applications including video indexing, motion comparison, and advanced biological study of visual signals in organisms. This dissertation presents an in-depth study of the development of several complexity measures for visual articulated motion. Optical flow is the basis of many motion estimation approaches and our first measure utilizes this as the starting point. Using optical flow, we develop a set of features to characterize different aspects of the motion and combine them to estimate the complexity of the movement. The second measure also utilizes optical flow, but uses higher-order features as motion descriptors. Specifically, features that encode the periodic nature of movements, synchrony, and movement clusters are developed and used toward the design of a new and improved complexity measure. To validate the measure, a human study was conducted. Subjects were asked to (a) give motion complexity scores to a set of videos and (b) rank features based on their importance to complexity. Using this study, we developed prediction models to estimate the motion complexity and also classification models to classify the videos. We use an alternative approach for our third measure based on interesting motion points in the combined space-time domain. These spatial-temporal interest points integrate hidden complexity information in the movement sequence. High level features are proposed to capture different dimensions of movement complexity from these interest points and then combined to estimate the overall complexity of the movement. All three approaches have been evaluated using two datasets: human movements and wolf spider movements. Extensive evaluation of the measures show the accuracy of estimating the complexity of articulated motion, and demonstrate the efficacy of their use toward classifying motion based on complexity. Adviser: Ashok Samal

Reactive motion planning and local navigation of robots remains a significant challenge in the motion control of robotic vehicles. This thesis presents new results on vision guided navigation using optical flow. By detecting key image features, calculating optical flow and leveraging time-to-transit (tau) as a feedback signal, control architectures can steer a vehicle so as to avoid obstacles while simultaneously using them as navigation beacons. Averaging and balancing tau over multiple image features successfully guides a vehicle along a corridor while avoiding looming objects in the periphery. In addition, the averaging strategy deemphasizes noise associated with rotationally induced flow fields, mitigating risks of positive feedback akin to the Larsen effect. A recently developed, biologically inspired, binary-key point description algorithm, FReaK, offers process speed-ups that make vision-based feedback signals achievable. A Parrot ARDrone2 has proven to be a reliable platform for testing the architecture and has demonstrated the control law's effectiveness in using time-to-transit calculations for real-time navigation.

This thesis presents new approaches to image registration which have low levels of computational complexity and their implementations on FPGAs. Firstly, a multi-pass image interpolation algorithm (MP-I2A) for image registration is proposed. By combining the optic flow estimation technique using the original image interpolation approach (I2A) with the least square estimate, the global motion between two images is estimated over several iterations. This new algorithm can estimate the affine motion model from the translation information computed for corresponding pairs of non-overlapping patches in two images. In addition to its insensitivity to noise, its performance is better than that of the Lucas-Kanade (LK) algorithm in terms of the number of iterations required for successful registration and the success rate. Secondly, a modified version of the MP-I2A, called the kernel-warping algorithm (KWA), is developed. It can estimate parameters of the affine motion model and requires fewer iterations than the original and LK algorithms which increases the feasibility of its implementation on FPGAs. A two-FPGA system is developed to implement it for the case of translation. Thirdly, the KWA is further developed to operate with binary (one-bit-per-pixel) images, whereby computations are much simpler than those of the original KWA as they use logic operations, such as AND, XOR, OR and NOT, instead of addition and multiplication. This significant reduction in computational complexity results in considerable savings in the hardware resources when implementing it on FPGAs. Finally, experiments to determine the optimal parameters for an adaptive low complexity global motion estimation algorithm, which are important for its successful implementation on FPGAs, are performed based on MATLAB simulations. The resulting FPGA implementation has improved processing speed and reduced hardware requirement when compare with those of previous approaches.

The problem of optical flow computation has various applications in Computer Vision, and serves as a key problem that has been well studied over the past decades. While most of the techniques for inferring optical flow are based on the brightness constancy assumption, various conditions including the presence of motion blur evidently violate this fundamental presumption. In low illumination scenarios and other conditions under which the shutter must be kept open for a relatively long interval, motion blur artifacts are inevitable. If the source image and the target image appear to be dissimilar due to different blur kernels, traditional methods will fail to achieve accurate results. After exploring advantages and shortcomings of various optical flow methods, e.g. CLG, Black-Anandan, and BlurFlow, we address the problem of optical flow in the presence of motion blur. In particular, we present a new approach that considers constructing a new pair of blurred frames, followed by regular optical flow computation. The proposed method, MB-CLG, eliminates the effect of non-uniform blur levels over the sequence. A proof is also provided to show the estimated flows are roughly equal to the ground truth flows that match the latent frames. The key observation is that if we applied the blur functions of the source image to the target image and vice versa, the brightness constancy assumption would be valid for the new frames. The proposed method employs a coarse-to-fine approach, in conjunction with a smoothness matrix to account for moving objects and occluded regions. Rather than warping frames or precomputing a large grid of derivatives as in Portz et al, MB-CLG directly warps the flows in the optimization process. This leads to lower computational cost, and requires less data storage. Based on the results for various synthetic sequences, MB-CLG outperforms existing optical flow algorithms in the sense of AAE, AEP and MSE.

With the increasing number of vehicles on the road the number of accidents have also been increasing. Development of various techniques such as lane departure warning systems that helps drivers to assist in driving can help reduce the number of accidents significantly. In this thesis, we attempt to develop such lane departure warning system by estimating the vehicle's lateral position. The lateral position of vehicle can be known if the heading angle of the vehicle can be determined. Therefore, this study focuses on determining heading angle and works toward development of the lane departure warning system based on image processing techniques. An in-vehicle camera is used to capture the images of the road in real time. The system then uses homography on this front - view images of the road to remove the perspective effect and transform the images such that the obtained resulting images represent as if they were observed from the top. The histogram equalization is also applied to the images to increase the global contrast by spreading out the most frequent intensity values. Shi and Tomasi corner detection technique has been used to find significant features (corners) in the images and Lucas - Kanade optical flow to track those corners in following images. The heading angle, and thus the lateral displacement, is determined by relating these tracked corners. As part of the study, a number of road tests were conducted on different roads of Duluth, MN and the findings based on the road tests are discussed.

In the context of extracting information from video, especially in the case of surveillance videos, bad weather conditions can pose a huge challenge. They affect feature extraction processes and hence the performance of other post-processing algorithms. In general, bad weather conditions can be classified into static and dynamic weather conditions. Static weather conditions like haze, fog and smoke cause blurring of features and saturation of intensities in the image. The temporal derivatives of the scene intensities are very low. Dynamic weather conditions like rain and snow have varying effects from frame to frame. The temporal derivative of the scene intensity for any pixel will not be zero in the presence of rain. In essence, the actual scene content is not occluded by rain or snow at all instances in the video sequence. In this research, a new framework is presented to achieve robust reconstruction of videos that are affected by rain. The main challenge is to model the location of rain streaks in a frame. This is due to the fact that the location of rain streaks at any particular instant is completely random. However, the changes in scene intensity caused by rain streaks have a generalized behavior. In addition, the instances in which the actual scene is not occluded is sufficient to enable modeling of an efficient technique to have a robust reconstruction of the scene. The first part of the proposed framework for rain removal is a novel technique to detect rain streaks based on phase congruency features. The features capture all structural edges that are conspicuous to the human visual system. The variation of features from frame to frame can be used to estimate the candidate rain pixels in a frame. In order to reduce the number of false candidates due to global motion, frames are registered using phase correlation. The presence of motion components in a local sense is ignored in this part of the framework. The second part of the proposed framework is a novel reconstruction technique that utilizes information from three different sources, which are intensities of the rain affected pixel, spatial neighbors, and temporal neighbors. An optimal estimate for the actual intensity of the rain affected pixel is made based on the minimization of registration error between frames. An optical flow technique based on local phase information is adopted for registration. This part of the proposed framework for removing rain is modeled such that the presence of local motion will not distort the features in the reconstructed video. The proposed framework is evaluated quantitatively and qualitatively on a variety of videos with varying complexities. The effectiveness of the algorithm is quantitatively verified by computing a no-reference image quality measure on individual frames of the reconstructed video. From a variety of experiments that are performed on output videos, it is shown that the proposed technique performs better than the state-of-the-art techniques. The performance of the proposed technique is evaluated in the case of removing snow from videos as well. It is observed that the method is capable of removing light snow streaks from the video. As part of ongoing research, attempts are being made at making the algorithm run in real-time.

A technique to analyze Particle Image Velocimetry (PIV) images with strong refractive distortion is presented. Based on a muti-level optical flow algorithm, the approach has been benchmarked with synthetically generated PIV images, and has been successfully applied to both experimental PIV and Laser Induced Fluorescence (LIF) images. To improve velocity field estimation, new symmetric spline and advection warping methods are proposed together with a data-driven method to impose boundary conditions. To address a historical weakness of optical flow PIV methods, a framework is presented to select the smoothness parameter for experimental images. An analysis of an experimental image series with severe blurring and intensity distortions demonstrates the viability of the technique.

Change detection is an ability that allows intelligent agents to react to unexpected situations. This mechanism is fundamental in providing more autonomy to robots. It has been used in many different fields including quality control and network intrusion. In the visual domain, however, most research has been confined to stationary cameras and only recently have researchers started to shift to mobile cameras. \ We propose a general framework for building internal spatial models of the visual experiences. These models are used to retrieve expectations about visual inputs which can be compared to the actual observation in order to identify the presence of changes. Our framework leverages the tolerance to small view changes of optic flow and color histogram representations and a self-organizing map to build a compact memory of camera observations. The effectiveness of the approach is demonstrated in a walking simulation, where spatial information and color histograms are combined to detect changes in a room. The location signal allows the algorithm to query the self-organizing map for the expected color histogram and compare it to the current input. Any deviations can be considered changes and are then localized on the input image. Furthermore, we show how detecting a vehicle entering or leaving the camera's lane can be reduced to a change detection problem. This simplifies the problem by removing the need to track or even know about other vehicles. Matching Pursuit is used to learn a compact dictionary to describe the observed experiences. Using this approach, changes are detected when the learned dictionary is unable to reconstruct the current input. The human experiments presented in this dissertation support the idea that humans build statistical models that evolve with experience. We provide evidence that not only does this experience improve people's behavior in 3D environments, but also enables them to detect chromatic changes. Mobile cameras are now part of our everyday lives, ranging from built-in laptop cameras to cell phone cameras. The vision of this research is to enable these devices with change detection mechanisms to solve a large class of problems. Beyond presenting a foundation that effectively detects changes in environments, we also show that the algorithms employed are computationally inexpensive. The practicality of this approach is demonstrated by a partial implementation of the algorithm on commodity hardware such as Android mobile devices.

This work describes a parallelizable optical flow estimator that uses a modified batch version of the Self Organizing Map (SOM). This gradient-based estimator handles the ill-posedness in motion estimation via a novel combination of regression and a self organization strategy. The aperture problem is explicitly modeled using an algebraic framework that partitions motion estimates obtained from regression into two sets, one (set Hc) with estimates with high confidence and another (set Hp) with low confidence estimates. The self organization step uses a uniquely designed pair of training set (Q=Hc) and the initial weights set (W=Hc U Hp). It is shown that with this specific choice of training and initial weights sets, the interpolation of flow vectors is achieved primarily due to the regularization property of SOM. Moreover, the computationally involved step of finding the winner unit in SOM simplifies to indexing into a 2D array making the algorithm parallelizable and highly scalable. To preserve flow discontinuities at occlusion boundaries, we have designed anisotropic neighborhood function for SOM that uses a novel OFCE residual-based distance measure. A multi-resolution or pyramidal approach is used to estimate large motion. As the algorithm is scalable, with sufficient number of computing cores (for example on a GPU), the implementation of the estimator can be made real-time. With the available true motion from Middlebury database, error metrics are computed.

We present a novel, real-time, markerless vision-based tracking system, employing a rigid orthogonal configuration of two pairs of opposing cameras. Our system uses optical flow over sparse features to overcome the limitation of vision-based systems that require markers or a pre-loaded model of the physical environment. We show how opposing cameras enable cancellation of common components of optical flow leading to an efficient tracking algorithm that captures five degrees of freedom including direction of translation and angular velocity. Experiments comparing our device with an electromagnetic tracker show that its average tracking accuracy is 80% over 185 frames, and it is able to track large range motions even in outdoor settings. We also present how opposing cameras in vision-based inside-looking-out systems can be used for gesture recognition. To demonstrate our approach, we discuss three different algorithms for recovering motion parameters at different levels of complete recovery. We show how optical flow in opposing cameras can be used to recover motion parameters of the multi-camera rig. Experimental results show gesture recognition accuracy of 88.0%, 90.7% and 86.7% for our three techniques, respectively, across a set of 15 gestures.

Abstract: Free viewpoint video (FVV) has been widely speculated as one of the next generation of visual media applications. By taking advantage of camera array based multiple imaging techniques, FVV enables free viewpoint navigation to invoke a sense of “being immersed” for the viewers. This thesis presents a cluster based FVV system which is designed as a specific application using a new proposed framework for general camera array applications. Our FVV system enables centralized workflow management and distributed computation to take advantage of the cluster’s computation power for fast FVV-oriented video processing. For its implementation, effort is mainly focused on the FVV workflow stages of multi-view video acquisition and dense depth based scene reconstruction. Specifically, a new automatic method is proposed for the efficient geometric, photometric and temporal calibrations of a camera array. With this novel integrated calibration method, the use of unsynchronized cameras becomes possible and the multi-view video acquisition is made easy, which greatly facilitate the practical use of a FVV (or camera array based) system. On the other hand, the dense depth based FVV scene reconstruction is addressed as an image discrete labeling problem using a novel coarse-to-fine region-tree based framework. As a general framework, its high ranking evaluations in standard binocular stereo matching and optical flow estimation benchmarking show its effectiveness and versatility. By further extending it for general position multi-view temporal stereo and integrating with inconsistency map/background based progressive optimization, spatial-temporal consistency is enforced in a new and unified way, which greatly helps the final FVV rendering quality. Extensive experimental results show that the new system with its accompanying algorithms can provide high quality rendering results.

A large number of image registration techniques have been developed for various types of sensors and applications, with the aim to improve the accuracy, computational complexity, generality, and robustness. They can be broadly classified into two categories: intensity-based and feature-based methods. The primary drawback of the intensity-based approaches is that it may fail unless the two images are misaligned by a moderate difference in scale, rotation, and translation. In addition, intensity-based methods lack the robustness in the presence of non-spatial distortions due to different imaging conditions between images. In this dissertation, the image registration is formulated as a two-stage hybrid approach combining both an initial matching and a final matching in a coarse-to-fine manner. In the proposed hybrid framework, the initial matching algorithm is applied at the coarsest scale of images, where the approximate transformation parameters could be first estimated. Subsequently, the robust gradient-based estimation algorithm is incorporated into the proposed hybrid approach using a multi-resolution scheme. Several novel and effective initial matching algorithms have been proposed for the first stage. The variations of the intensity characteristics between images may be large and non-uniform because of non-spatial distortions. Therefore, in order to effectively incorporate the gradient-based robust estimation into our proposed framework, one of the fundamental questions should be addressed: what is a good image representation to work with using gradient-based robust estimation under non-spatial distortions. With the initial matching algorithms applied at the highest level of decomposition, the proposed hybrid approach exhibits superior range of convergence. The gradient-based algorithms in the second stage yield a robust solution that precisely registers images with sub-pixel accuracy. A hierarchical iterative searching further enhances the convergence range and rate. The simulation results demonstrated that the proposed techniques provide significant benefits to the performance of image registration.

Motion in depth and/or zooming causes defocus blur. This work presents a solution to the problem of using defocus blur and optical flow information to compute depth at points that defocus when they move. We first formulate a novel algorithm which recovers defocus blur and affine parameters simultaneously. Next we formulate a novel relationship (the blur-depth relationship) between defocus blur, relative object depth and three parameters based on camera motion and intrinsic camera parameters. We can handle the situation where a single image has points which have defocused, got sharper or are focally unperturbed. Moreover, our formulation is valid regardless of whether the defocus is due to the image plane being in front of or behind the point of sharp focus.The blur-depth relationship requires a sequence of at least three images taken with the camera moving either towards or away from the object. It can be used to obtain an initial estimate of relative depth using one of several non-linear methods. We demonstrate a solution based on the Extended Kalman Filter in which the measurement equation is the blur-depth relationship. The estimate of relative depth is then used to compute an initial estimate of camera motion parameters. In order to refine depth values, the values of relative depth and camera motion are then input into a second Extended Kalman Filter in which the measurement equations are the discrete motion equations. This set of cascaded Kalman filters can be employed iteratively over a longer sequence of images in order to further refine depth. We conduct several experiments on real scenery in order to demonstrate the range of object shapes that the algorithm can handle. We show that fairly good estimates of depth can be obtained with just three images.

Determining optical flow is a fundamental problem in computer vision and image analysis. Optical flow is the projection of three dimensional motion of objects onto a plane, estimated from a sequence of two dimensional images. However, as this is an ill posed problem, i.e. has no unique solution based on available information thus requiring additional constraints to make the problem well posed. We propose an algorithm based on feature matching using Hopfield Networks to determine the optical flow field between a sequence of images. We formulate the correspondence between features extracted from the two images as an optimization problem using a new robust disparity measure. The Hopfield network is then used to minimize the cost function and obtain an optimal match between the features in successive frames. This thesis offers a review of relevant literature and a detailed description of the algorithm. We then evaluate the performance of the algorithm on a variety of test sequences, both artificial and real. The results are found to be comparable to those obtained by other feature based algorithms, which suggests the use of Hopfield type networks in practical vision algorithms.

Given 2D images, it still remains a big challenge in the field of computer vision to group the image points into logical objects (segmentation) and to determine the locations in the scene (range estimation). Despite the decades of research, a single solution is yet to be found. Through our research we have demonstrated that a possible solution is to use moving aperture lens. This lens has the effect of introducing small, repeating movements of the camera center so that objects appear to translate in the image, by an amount that depends on distance from the plane of focus. Our novel method employs optical flow techniques to an image sequence, captured using a video camera with a moving aperture lens. For a stationary scene, optical flow magnitudes and direction are directly related to the three-dimensional object distance and location from the observer. Exploiting this information, we have successfully extracted objects at different depths and estimated the locations of objects in the scene, with respect to the plane of focus. Our work therefore demonstrates an ability for passive range estimation, without emitting any energy in an environment. Other potential applications include video compression, 3D video broadcast, teleconferencing and autonomous vehicle navigation.