Search

Visual-based navigation systems for Unmanned Aerial vehicles (UAVs) have become an interesting research area focused on improving robustness and accuracy in the urban environment. However, a lack of integrity can damage UAVs, limiting their potential usage in civil applications. For safety reasons, integrity performance requirements must be met. In literature, such systems require significant attention for their ability to perform fault analysis, referred to as failure mode. In this paper, we have conducted a failure mode analysis in urban environments for UAVs to identify threats and faults presented in existing Visual-inertial Navigation Systems. In addition, we propose a federated-filter-based fault detection and execution system to improve navigation performance under faulted conditions.

Among the main challenges in robotics, target-driven visual navigation has gained increasing interest in recent years. In this task, an agent has to navigate in an environment to reach a user specified target, only through vision. Recent fruitful approaches rely on deep reinforcement learning, which has proven to be an effective framework to learn navigation policies. However, current state-of-the-art methods require to retrain, or at least fine-tune, the model for every new environment and object. In real scenarios, this operation can be extremely challenging or even dangerous. For these reasons, we address generalization in target-driven visual navigation by proposing a novel architecture composed of two networks, both exclusively trained in simulation. The first one has the objective of exploring the environment, while the other one of locating the target. They are specifically designed to work together, while separately trained to help generalization. In this article, we test our agent in both simulated and real scenarios, and validate its generalization capabilities through extensive experiments with previously unseen goals and unknown mazes, even much larger than the ones used for training.

Endobronchial intervention is increasingly used as a minimally invasive means of lung intervention. Vision-based localization approaches are often sensitive to image artifacts in bronchoscopic videos. In this letter, a robust navigation system based on a context-aware depth recovery approach for monocular video images is presented. To handle the artifacts, a conditional generative adversarial learning framework is proposed for reliable depth recovery. The accuracy of depth estimation and camera localization is validated on an in vivo dataset. Both quantitative and qualitative results demonstrate that the depth recovered with the proposed method preserves better structural information of airway lumens in the presence of image artifacts, and the improved camera localization accuracy demonstrates its clinical potential for bronchoscopic navigation.

Appearance changes are one of the most challenging problems for the visual localization of an autonomous vehicle in outdoor environments. Data association between the current image and the landmarks in the map can be difficult if the map was built with different environmental conditions. The work developed in this PhD thesis is focused on life-long navigation for autonomous shuttles in outdoor environments where the appearance is subject to substantial changes.In this thesis, we put forward several contributions to improve localization robustness and accuracy in long-term scenarios. The proposed approaches present a solution to build and use multi-session maps that incorporate sequences recorded in different conditions (day, night, fog, snow, rain, change of season, etc.). We can categorize our proposed approaches in two ways: keyframes retrieval and map management.The keyframes retrieval process is carried out online during visual localization, in which, we take advantage of a visual landmark map composed of N sequences gathered at different times and conditions. During each localization session, we exploit information collected in the beginning of the trajectory to compute a ranking function which will be used in the rest of the trajectory to retrieve from the map the keyframes that maximise the number of matched points. The retrieval depends on the geometric distance between the pose of the keyframe and the current pose of the vehicle, and the similarity of this keyframe with the current environmental condition. In the mapping phase, covering all conditions by constantly adding data to the map leads to a continuous growth in the map size which in turn deteriorates the localization speed and performance. Therefore, we employ a map management phase which has the aim of maintaining a long-term reliable map with a fixed size throughout the frequent runs. To efficiently reduce the size of the map without significantly degrading the localization performance, we tend to remove superfluous data from the map. Accordingly, we have proposed different map management strategies by using different kind of information, such as the traversals similarity information computed during the keyframes retrieval process or the sun's position information related to each traversal, to analyse the data redundancy in the map.Throughout this report, we provide the results of our experiments which were performed on real data collected with our autonomous shuttle as well as on a widely used public dataset. By comparing our proposed keyframes retrieval technique with a simple strategy where the keyframes are retrieved only according to their geometric distance to the vehicle pose, we have demonstrated that our approach has significantly improved localization performance in different challenging conditions. On several aspects related to map management, we demonstrated that our proposed strategies have outperformed a state-of-the-art map management approach.We have also presented a new dataset that contains challenging environmental conditions which we make available to the community in the hope that it will be useful to other people working in this field., Les changements d'apparence sont l'un des problèmes les plus difficiles pour la localisation visuelle d'un véhicule autonome dans des environnements extérieurs. L'association des données entre l'image actuelle et les amers dans la carte peut être difficile si la carte a été construite dans des conditions environnementales différentes. Les travaux développés dans cette thèse de doctorat se concentrent sur la localisation à long terme pour les navettes autonomes dans des environnements extérieurs où l'apparence est soumise à des changements substantiels. Dans nos travaux, nous proposons plusieurs contributions pour améliorer la robustesse et la précision de la localisation dans des scénarios à long terme. Les approches proposées présentent une solution pour construire et utiliser des cartes multi-sessions qui intègrent des séquences enregistrées dans différentes conditions (jour, nuit, brouillard, neige, pluie, changement de saison, etc.). Nous pouvons catégoriser nos approches proposées en deux volets : la récupération des images clés et la gestion des cartes. Le processus de récupération des images clés est effectué en ligne lors de la localisation visuelle, dans laquelle nous tirons avantage d'une carte d’amers visuels composée de N séquences rassemblées à des moments et dans des conditions différents. Lors de chaque session de localisation, nous exploitons les informations collectées au début de la trajectoire pour calculer une fonction de classement (ranking function) qui sera utilisée dans le reste de la trajectoire pour récupérer de la carte les images clés qui maximisent le nombre de points appariés. La récupération dépend de la distance géométrique entre la pose de l'image clé et la pose actuelle du véhicule, et de la similitude de cette image clé avec les conditions environnementales actuelles. Dans la phase de cartographie, couvrir toutes les conditions en ajoutant constamment des données à la carte conduit à une croissance continue de la taille de la carte qui à son tour détériore la vitesse et les performances de localisation. Par conséquent, nous utilisons une phase de gestion de la carte qui a pour objectif de maintenir une carte fiable à long terme avec une taille fixe tout au long des courses fréquentes. Pour réduire efficacement la taille de la carte sans dégrader considérablement les performances de localisation, nous avons tendance à supprimer les données superflues. En conséquence, nous avons proposé différentes stratégies de gestion de carte en utilisant différents types d'informations, telles que les informations de similarité entre les traversées calculées pendant le processus de récupération des images clés ou les informations de position du soleil liées à chaque traversée, pour analyser la redondance des données dans la carte. Les résultats de nos expériences ont été réalisées sur des données réelles collectées avec notre navette autonome ainsi que sur un jeu de données publique largement utilisé. En comparant notre technique de récupération d'images clés proposée avec une stratégie primitive où les images clés sont récupérées uniquement en fonction de leur distance géométrique à la pose du véhicule, nous avons démontré que notre approche a considérablement amélioré la performance de localisation dans différentes conditions difficiles. Sur plusieurs aspects liés à la gestion des cartes, nous avons démontré que nos stratégies proposées ont surpassé une approche de gestion de carte proposée dans l’état de l’art. Nous avons également présenté un nouvel ensemble de données contenant des conditions environnementales difficiles que nous avons mis à la disposition de la communauté dans l'espoir qu'il sera utile à d'autres personnes travaillant dans ce domaine.

The ability to recognize previously mapped locations is an essential feature for autonomous systems. Unstructured planetary-like environments pose a major challenge to these systems due to the similarity of the terrain. As a result, the ambiguity of the visual appearance makes state-of-the-art visual place recognition approaches less effective than in urban or man-made environments. This paper presents a method to solve the loop closure problem using only spatial information. The key idea is to use a novel continuous and probabilistic representations of terrain elevation maps. Given 3D point clouds of the environment, the proposed approach exploits Gaussian Process (GP) regression with linear operators to generate continuous gradient maps of the terrain elevation information. Traditional image registration techniques are then used to search for potential matches. Loop closures are verified by leveraging both the spatial characteristic of the elevation maps (SE(2) registration) and the probabilistic nature of the GP representation. A submap-based localization and mapping framework is used to demonstrate the validity of the proposed approach. The performance of this pipeline is evaluated and benchmarked using real data from a rover that is equipped with a stereo camera and navigates in challenging, unstructured planetary-like environments in Morocco and on Mt. Etna., This work is accepted for presentation at the 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). Please find IEEE's copyright statement at the bottom of the first page. Cedric Le Gentil and Mallikarjuna Vayugundla share the first authorship of this paper

This paper fosters the idea that deep learning methods can be used to complement classical visual odometry pipelines to improve their accuracy and to associate uncertainty models to their estimations. We show that the biases inherent to the visual odometry process can be faithfully learned and compensated for, and that a learning architecture associated with a probabilistic loss function can jointly estimate a full covariance matrix of the residual errors, defining an error model capturing the heteroscedasticity of the process. Experiments on autonomous driving image sequences assess the possibility to concurrently improve visual odometry and estimate an error associated with its outputs., Accepted in IEEE Robotics and Automation Letters and IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2020

This work proposes a novel deep network architecture to solve the camera ego-motion estimation problem. A motion estimation network generally learns features similar to optical flow (OF) fields starting from sequences of images. This OF can be described by a lower dimensional latent space. Previous research has shown how to find linear approximations of this space. We propose to use an autoencoder network to find a nonlinear representation of the OF manifold. In addition, we propose to learn the latent space jointly with the estimation task, so that the learned OF features become a more robust description of the OF input. We call this novel architecture latent space visual odometry (LS-VO). The experiments show that LS-VO achieves a considerable increase in performances with respect to baselines, while the number of parameters of the estimation network only slightly increases.

In this work, we address the problem of visual navigation by following instructions. In this task, the robot must interpret a natural language instruction in order to follow a predefined path in a possibly unknown environment. Despite different approaches have been proposed in the last years, they are all based on the assumption that the environment contains objects or other elements that can be used to formulate instructions, such as houses or offices. On the contrary, we focus on situations where the environment objects cannot be used to specify a navigation path. In particular, we consider 3D maze-like environments as our test bench because they can be very large and offer very intricate structures. We show that without reference points, visual navigation and instruction following can be rather challenging, and that standard approaches can not be applied successfully. For this reason, we propose a new architecture that explicitly learns both visual navigation and instruction understanding. We demonstrate with simulated experiments that our method can effectively follow instructions and navigate in previously unseen mazes of various sizes.

International audience; This paper considers the problem of collision-free navigation of omnidirectional mobile robots in environments with obstacles. Information from a monocular camera, encoders, and an inertial measurement unit is used to achieve the task. Three different visual servoing control schemes, compatible with the class of considered robot kinematics and sensor equipment, are analysed and their robustness properties with respect to actuation inaccuracies discussed. Then, a controller is proposed with formal guarantee of convergence to the bisector of a corridor. The main controller components are a visual servoing control scheme and a velocity estimation algorithm integrating visual, kinematic and inertial information. The behaviour of all the considered algorithms is analised and illustrated through simulations both for a wheeled and a humanoid robot. The solution proposed as the most efficient and robust with respect to actuation inaccuracies is also validated experimentally on a real humanoid NAO.

Robots must reliably interact with refractive objects in many applications; however, refractive objects can cause many robotic vision algorithms to become unreliable or even fail, particularly feature-based matching applications, such as structure-from-motion. We propose a method to distinguish between refracted and Lambertian image features using a light field camera. Specifically, we propose to use textural cross-correlation to characterise apparent feature motion in a single light field, and compare this motion to its Lambertian equivalent based on 4D light field geometry. Our refracted feature distinguisher has a 34.3% higher rate of detection compared to state-of-the-art for light fields captured with large baselines relative to the refractive object. Our method also applies to light field cameras with much smaller baselines than previously considered, yielding up to 2 times better detection for 2D-refractive objects, such as a sphere, and up to 8 times better for 1D-refractive objects, such as a cylinder. For structure from motion, we demonstrate that rejecting refracted features using our distinguisher yields up to 42.4% lower reprojection error, and lower failure rate when the robot is approaching refractive objects. Our method lead to more robust robot vision in the presence of refractive objects., Comment: 8 pages, 8 figures, submission to IROS 2018

Rover is a robotic system that integrates a simple system implementing electrical and mechanical components together. In this study, we propose a rover using mechanical components which consist of a robotic arm, joint and mechanical gripper, backbone chassis and continues track, while the electrical components include servo motor, servo controller, transmitter and receiver for vision system and wireless controller via USB host as its control system. The purpose of this project is for monitoring and safety purposes. In addition, the main goal of this project is to develop a simple robotic rover that is easy to build and manufacture as well as cost-effective. To add more functionality on this rover, it is equipped with a robotic arm and real-time view camera integration. This rover is equipped with a first-person view (FPV) camera, an integrated camera on the rover that can give clear visibility and direction to the rover pilot. The live feed can be viewed on the monitor inside the command station box. It can be used to assist safety authorities to collect information & insights, work lift to collect and remove the load and to conduct search and rescue operation. As for the result, the mobility system of the robotic rover at terrain surfaces and analyses the capabilities of the chassis during lifting load had been tested.

This work addresses the problem of coupling vision-based navigation systems for Unmanned Aerial Vehicles (UAVs) with robust obstacle avoidance capabilities. The former problem is solved by maximizing the visibility of the points of interest, while the latter is modeled by means of ellipsoidal repulsive areas. The whole problem is transcribed into an Optimal Control Problem (OCP), and solved in a few milliseconds by leveraging state-of-the-art numerical optimization. The resulting trajectories are well suited for reaching the specified goal location while avoiding obstacles with a safety margin and minimizing the probability of losing the route with the target of interest. Combining this technique with a proper ellipsoid shaping (i.e., by augmenting the shape proportionally with the obstacle velocity or with the obstacle detection uncertainties) results in a robust obstacle avoidance behavior. We validate our approach within extensive simulated experiments that show effective capabilities to satisfy all the constraints even in challenging conditions. We release with this paper the open source implementation.

Place recognition is an essential capability for robotic autonomy. While ground robots observe the world from generally similar viewpoints over repeated visits, other robots, such as small aircraft, experience far more different viewpoints, requiring place recognition for images captured from very wide baselines. While traditional feature-based methods fail dramatically under extreme viewpoint changes, deep learning approaches demand heavy runtime processing. Driven by the need for cheaper alternatives able to run on computationally restricted platforms, such as small aircraft, this work proposes a novel real-time pipeline employing depth-completion on sparse feature maps that are anyway computed during robot localization and mapping, to enable place recognition at extreme viewpoint changes. The proposed approach demonstrates unprecedented precision-recall rates on challenging benchmarking and own synthetic and real datasets with up to 45◦ difference in viewpoints. In particular, our synthetic datasets are, to the best of our knowledge, the first to isolate the challenge of viewpoint changes for place recognition, addressing a crucial gap in the literature. All of the new datasets are publicly available to aid benchmarking., IEEE Robotics and Automation Letters, 4 (2), ISSN:2377-3766

International audience; This letter analyzes the visual–inertial sensor fusion problem in the cooperative case of two agents, and proves that this sensor fusion problem is equivalent to a simple polynomial equations system that consists of several linear equations and three polynomial equations of second degree. The analytic solution of this polynomial equations system is easily obtained by using an algebraic method. In other words, this letter provides the analytic solution to the visual–inertial sensor fusion problem in the case of two agents. The power of the analytic solution is twofold. From one side, it allows us to determine the relative state between the agents (i.e., relative position, speed, and orientation) without the need of an initialization. From another side, it provides fundamental insights into all the theoretical aspects of the problem. This letter mainly focuses on the first issue. However, the analytic solution is also exploited to obtain basic structural properties of the problem that characterize the observability of the absolute scale and the relative orientation. Extensive simulations and real experiments show that the solution is successful in terms of precision and robustness.

Vision constitutes one of the most important cues in robotics. A single monocular camera can provide rich visual information at a reasonable rate that can be used as a feedback for control, state estimation of mobile robots or safe navigation in unknown environments for instance. However, it is necessary to satisfy particular visual constraints on the image such as visibility and occlusion constraints during motion to keep some visual targets visible. Quadrotors are endowed with very reactive motion capabilities due to their compact structure and motor configuration. Moreover, vision from a (fixed) on-board camera will suffer from rotation motions due to the system underactuation. In this thesis, we want to benefit from the system aggressiveness to perform several vision-based navigation tasks. We assume state estimation relies solely on sensor fusion of an onboard inertial measurement unit (IMU) and a monocular camera that provides reliable pose estimates. Therefore, visual constraints are challenging and critical in this context. In this thesis we exploit numerical optimization to design feasible trajectories satisfying several state, input and visual nonlinear constraints. With the help of differential flatness and B-spline parametrization we will propose an efficient replanning strategy inspired form Model Predictive Control to generate smooth and agile trajectories. Finally, we propose a minimum-time planning algorithm that handles intermittent visibility losses in order to navigate in larger cluttered environments. This contribution brings state estimation uncertainty at the planning stage to produce robust and safe trajectories. All the theoretical developments discussed in this thesis are corroborated by simulations and experiments run by using a quadrotor UAV. The reported results show the effectiveness of proposed techniques.; La vision représente un des plus importants signaux en robotique. Une caméra monoculaire peut fournir de riches informations visuelles à une fréquence raisonnable pouvant être utilisées pour la commande, l’estimation d’état ou la navigation dans des environnements inconnus par exemple. Il est cependant nécessaire de respecter des contraintes visuelles spécifiques telles que la visibilité de mesures images et les occultations durant le mouvement afin de garder certaines cibles visuelles dans le champ de vision. Les quadrirotors sont dotés de capacités de mouvement très réactives du fait de leur structure compacte et de la configuration des moteurs. De plus, la vision par une caméra embarquée (fixe) va subir des rotations dues au sous-actionnement du système. Dans cette thèsenous voulons bénéficier de l’agilité du quadrirotor pour réaliser plusieurs tâches de navigation basées vision. Nous supposons que l’estimation d’état repose uniquement sur la fusion capteurs d’une centrale inertielle (IMU) et d’une caméra monoculaire qui fournit des estimations de pose précises. Les contraintes visuelles sont donc critiques et difficiles dans un tel contexte. Dans cette thèse nous exploitons l’optimisation numérique pour générer des trajectoires faisables satisfaisant un certain nombre de contraintes d’état, d’entrées et visuelles non linéaires. A l’aide la platitude différentielle et de la paramétrisation par des B-splines nous proposons une stratégie de replanification performante inspirée de la commande prédictive pour générer des trajectoires lisses et agiles. Enfin, nous présentons un algorithme de planification en temps minimum qui supporte des pertes de visibilité intermittentes afin de naviguer dans des environnements encombrés plus vastes. Cette contribution porte l’incertitude de l’estimation d’état au niveau de la planification pour produire des trajectoires robustes et sûres. Les développements théoriques discutés dans cette thèse sont corroborés par des simulations et expériences en utilisant un quadrirotor. Les résultats reportés montrent l’efficacité des techniques proposées.

In this work, a novel monocular simultaneous localization and mapping (SLAM) system with application to micro aerial vehicles is proposed. The main difference with respect to previous approaches is that a barometer is used as a unique sensory aid for incorporating altitude information into the system in order to recover an absolute metric scale. First, an observability analysis of a simplified model of a monocular SLAM system is developed. From this analysis, several theoretical results are derived. Among others, one important result is related to the fact that the metric scale can become observable when measurements of altitude are included in the system. In this case, sufficient conditions for observability are presented. The design of the proposed method is based on these theoretical results. Simulations and experiments with real data are presented to validate the proposed approach. The results confirm that the metric scale can be retrieved by including altitude measurements in the system. It is also shown that the proposed method can be practically implemented, using low-cost sensors, to perform visual-based navigation in GPS-denied environments.