Search

We present a method for computing egomotion using event cameras with a pre-trained optical flow spiking neural network (SNN). To address the aperture problem encountered in the sparse and noisy normal flow of the initial SNN layers, our method includes a sliding-window bin-based pooling layer that computes a fused full flow estimate. To add robustness to noisy flow estimates, instead of computing the egomotion from vector averages, our method optimizes the intersection of constraints. The method also includes a RANSAC step to robustly deal with outlier flow estimates in the pooling layer. We validate our approach on both simulated and real scenes and compare our results favorably to the state-of-the-art methods. However, our method may be sensitive to datasets and motion speeds different from those used for training, limiting its generalizability., This work received support from projects EBCON (PID2020-119244GBI00) and AUDEL (TED2021-131759A-I00) funded by MCIN/ AEI/ 10.13039/ 501100011033 and by the "European Union NextGenerationEU/PRTR"; the Consolidated Research Group RAIG (2021 SGR 00510) of the Departament de Recerca i Universitats de la Generalitat de Catalunya; and by an FI AGAUR PhD grant to Yi Tian., Peer Reviewed, Postprint (author's final draft)

(English) Event-based cameras are novel sensors with a bio-inspired design that exhibit a high dynamic range and extremely low latency. They sensing principle is different than the one of conventional cameras. These novel sensors generate asynchronous streams of events triggered proportionally to the logarithmic pixel-level change of brightness. Each event is timestamped with a micro-second resolution and transmitted as soon as it is fired. Thus, event cameras convey non-redundant information and capture fast motion. These assets make them suitable for high dynamics and challenging lighting applications without image blurring, over or underexposure problems. The event camera's operational principle requires new algorithms to deal with image intensity's absence and leverage its fast asynchronous response. This thesis explores new event-based monocular solutions to the localization and mapping problem (SLAM) in human-made scenarios. Concerning the event camera's natural edge response, the use of line features was evaluated throughout the chapters of this thesis. A high-speed camera pose tracking approach was proposed that exploits the fast response of an event camera to accurately estimate its 6 degrees of freedom. Lines are efficiently matched to the events with a proposed fast data-association mechanism. In the tracking formulation, event-per-event and window of events representations were analyzed to identify their advantages and real-time implementation feasibility without introducing speed bottlenecks. Additionally, several estimation variants were implemented regarding measurement and motion models. This thesis also analyzes sensor fusion methodologies to combine events with inertial data. The inertial measurements come at a lower rate than events; thus, the objective is to avoid conventional inertial integration and cooperate with the event data to correct state parameters. The sensorial fusion allowed more accurate estimations but slowed down and constrained t, (Español) Les càmeres d'esdeveniments són sensors nous amb un disseny d'inspiració biològica que presenten un rang dinàmic elevat i una latència baixa. El seu principi de detecció és diferent al de les càmeres convencionals. Aquests sensors generen paquets asíncrons d'esdeveniments produïts pel canvi logarítmic de la brillantor a nivell de píxels. Cada esdeveniment té un temps amb una resolució de microsegons i es transmet tan bon punt s'activa. Així, les càmeres d'esdeveniments no transmeten informació redundant i capturen moviment ràpids. Això els fa adequats per a aplicacions d'alta dinàmica i d'il·luminació desafiants sense problemes de desenfocament de la imatge, sobre o subexposició. El principi de funcionament de la càmera d'esdeveniments requereix nous algorismes per fer front a l'absència de la intensitat de la imatge i aprofitar la seva resposta ràpida. Aquesta tesi explora noves solucions monoculars amb esdeveniments al problema de localització i mapeig (SLAM) en escenaris creats per l'home. Pel que fa a la resposta de la vora natural de la càmera d'esdeveniments, es va avaluar l'ús de línies. Es va proposar una metodologia per estimar els 6 graus de llibertat de la càmera per en dinàmiques altes on les línies estan relacionades amb els esdeveniments amb un mecanisme ràpid d'associació. Durant la formulació de la localització, es van analitzar les representacions d'esdeveniment per esdeveniment i finestra d'esdeveniments per identificar els seus avantatges i la viabilitat d'implementació en temps realt. A més, es van implementar diverses variants d'estimació pel que fa als models de mesura i moviment. Aquesta tesi també analitza metodologies de fusió sensorial per combinar esdeveniments amb IMU. Les mesures inercials arriben a un ritme inferior a les estimacions d'esdeveniments; així, l'objectiu és cooperar amb les dades de l'esdeveniment per corregir l'estat en diferents moments. La fusió sensorial va permetre estimacions més precises, però va frenar i v, Postprint (published version)

This study presents new methods for integrating event data and IMU readings to achieve ultra-fast camera pose estimates. The conventional predict-with-IMU-correct-with-vision approach is no longer optimal because events can be generated much more rapidly than IMU data. Therefore, two novel fusion schemes are proposed, which combine constant velocity and constant acceleration prediction models with ultra-fast (10 kHz) event-based updates and slower 1 kHz IMU updates. The first scheme uses IMU data as instantaneous measurements of acceleration and angular rate, while the second scheme considers these measurements as the average within the IMU sampling time. To provide a basis for comparison, the traditional method that predicts motion using IMU data and updates the estimates with event-feature matching over a 1ms time window is also implemented. All models are designed as Kalman filter variants, which act as the tracker module of a PTAM system in a human-made indoor scenario and are subjected to stress experiments to evaluate their capabilities. The models are also compared against an event-only estimator and a frame-based visual-inertial approach. The findings demonstrate superior performance at a throughput that is 100 times faster than the state-of-the-art.

We present a method for computing egomotion using event cameras with a pre-trained optical flow spiking neural network (SNN). To address the aperture problem encountered in the sparse and noisy normal flow of the initial SNN layers, our method includes a sliding-window bin-based pooling layer that computes a fused full flow estimate. To add robustness to noisy flow estimates, instead of computing the egomotion from vector averages, our method optimizes the intersection of constraints. The method also includes a RANSAC step to robustly deal with outlier flow estimates in the pooling layer. We validate our approach on both simulated and real scenes and compare our results favorably to the state-of-the-art methods. However, our method may be sensitive to datasets and motion speeds different from those used for training, limiting its generalizability.

Event-based cameras are novel sensors with a bio-inspired design that exhibit a high dynamic range and extremely low latency. They sensing principle is different than the one of conventional cameras. These novel sensors generate asynchronous streams of events triggered proportionally to the logarithmic pixel-level change of brightness. Each event is timestamped with a micro-second resolution and transmitted as soon as it is fired. Thus, event cameras convey non-redundant information and capture fast motion. These assets make them suitable for high dynamics and challenging lighting applications without image blurring, over or underexposure problems. The event camera's operational principle requires new algorithms to deal with image intensity's absence and leverage its fast asynchronous response. This thesis explores new event-based monocular solutions to the localization and mapping problem (SLAM) in human-made scenarios. Concerning the event camera's natural edge response, the use of line features was evaluated throughout the chapters of this thesis. A high-speed camera pose tracking approach was proposed that exploits the fast response of an event camera to accurately estimate its 6 degrees of freedom. Lines are efficiently matched to the events with a proposed fast data-association mechanism. In the tracking formulation, event-per-event and window of events representations were analyzed to identify their advantages and real-time implementation feasibility without introducing speed bottlenecks. Additionally, several estimation variants were implemented regarding measurement and motion models. This thesis also analyzes sensor fusion methodologies to combine events with inertial data. The inertial measurements come at a lower rate than events; thus, the objective is to avoid conventional inertial integration and cooperate with the event data to correct state parameters. The sensorial fusion allowed more accurate estimations but slowed down and constrained the motion