Search

Autonomous space rovers face significant challenges when navigating deformable and heterogeneous terrains due to variability in soil properties, which can lead to severe wheel slip, compromising navigation efficiency and increasing the risk of entrapment. To address this problem, we introduce SlipNet, a novel approach for predicting wheel slip in segmented regions of diverse terrain surfaces without relying on prior terrain classification. SlipNet employs dynamic terrain segmentation and slip assignment techniques on previously unseen data, enhancing rover navigation capabilities in uncertain environments. We developed a synthetic data generation framework using the high-fidelity Vortex Studio simulator to create realistic datasets that replicate a wide range of deformable terrain conditions for training and evaluation. Extensive simulation results demonstrate that our model, combining DeepLab v3+ with SlipNet, significantly outperforms the state-of-the-art TerrainNet method, achieving lower mean absolute error (MAE) across five distinct terrain samples. These findings highlight the effectiveness of SlipNet in improving rover navigation in challenging terrains.

Moving object segmentation is critical to interpret scene dynamics for robotic navigation systems in challenging environments. Neuromorphic vision sensors are tailored for motion perception due to their asynchronous nature, high temporal resolution, and reduced power consumption. However, their unconventional output requires novel perception paradigms to leverage their spatially sparse and temporally dense nature. In this work, we propose a novel event-based motion segmentation algorithm using a Graph Transformer Neural Network, dubbed GTNN. Our proposed algorithm processes event streams as 3D graphs by a series of nonlinear transformations to unveil local and global spatiotemporal correlations between events. Based on these correlations, events belonging to moving objects are segmented from the background without prior knowledge of the dynamic scene geometry. The algorithm is trained on publicly available datasets including MOD, EV-IMO, and \textcolor{black}{EV-IMO2} using the proposed training scheme to facilitate efficient training on extensive datasets. Moreover, we introduce the Dynamic Object Mask-aware Event Labeling (DOMEL) approach for generating approximate ground-truth labels for event-based motion segmentation datasets. We use DOMEL to label our own recorded Event dataset for Motion Segmentation (EMS-DOMEL), which we release to the public for further research and benchmarking. Rigorous experiments are conducted on several unseen publicly-available datasets where the results revealed that GTNN outperforms state-of-the-art methods in the presence of dynamic background variations, motion patterns, and multiple dynamic objects with varying sizes and velocities. GTNN achieves significant performance gains with an average increase of 9.4% and 4.5% in terms of motion segmentation accuracy (IoU%) and detection rate (DR%), respectively.

Robotic grippers are receiving increasing attention in various industries as essential components of robots for interacting and manipulating objects. While significant progress has been made in the past, conventional rigid grippers still have limitations in handling irregular objects and can damage fragile objects. We have shown that soft grippers offer deformability to adapt to a variety of object shapes and maximize object protection. At the same time, dynamic vision sensors (e.g., event-based cameras) are capable of capturing small changes in brightness and streaming them asynchronously as events, unlike RGB cameras, which do not perform well in low-light and fast-moving environments. In this paper, a dynamic-vision-based algorithm is proposed to measure the force applied to the gripper. In particular, we first set up a DVXplorer Lite series event camera to capture twenty-five sets of event data. Second, motivated by the impressive performance of the Vision Transformer (ViT) algorithm in dense image prediction tasks, we propose a new approach that demonstrates the potential for real-time force estimation and meets the requirements of real-world scenarios. We extensively evaluate the proposed algorithm on a wide range of scenarios and settings, and show that it consistently outperforms recent approaches., Comment: 6 pages, 5 figures

Tactile sensing represents a crucial technique that can enhance the performance of robotic manipulators in various tasks. This work presents a novel bioinspired neuromorphic vision-based tactile sensor that uses an event-based camera to quickly capture and convey information about the interactions between robotic manipulators and their environment. The camera in the sensor observes the deformation of a flexible skin manufactured from a cheap and accessible 3D printed material, whereas a 3D printed rigid casing houses the components of the sensor together. The sensor is tested in a grasping stage classification task involving several objects using a data-driven learning-based approach. The results show that the proposed approach enables the sensor to detect pressing and slip incidents within a speed of 2 ms. The fast tactile perception properties of the proposed sensor makes it an ideal candidate for safe grasping of different objects in industries that involve high-speed pick-and-place operations., Comment: 9 pages, 10 figures, journal

Bilateral teleoperation of low-speed Unmanned Ground Vehicles (UGVs) on soft terrains is crucial for applications like lunar exploration, offering effective control of terrain-induced longitudinal slippage. However, latency arising from transmission delays over a network presents a challenge in maintaining high-fidelity closed-loop integration, potentially hindering UGV controls and leading to poor command-tracking performance. To address this challenge, this paper proposes a novel predictor framework that employs a Physics-informed Long Short-Term Memory (PiLSTM) network for designing bilateral teleoperator controls that effectively compensate for large delays. Contrasting with conventional model-free predictor frameworks, which are limited by their linear nature in capturing nonlinear and temporal dynamic behaviors, our approach integrates the LSTM structure with physical constraints for enhanced performance and better generalization across varied scenarios. Specifically, four distinct predictors were employed in the framework: two compensate for forward delays, while the other two compensate for backward delays. Due to their effectiveness in learning from temporal data, the proposed PiLSTM framework demonstrates a 26.1\ improvement in delay compensation over the conventional model-free predictors for large delays in open-loop case studies. Subsequently, experiments were conducted to validate the efficacy of the framework in close-loop scenarios, particularly to compensate for the real-network delays experienced by teleoperated UGVs coupled with longitudinal slippage. The results confirm the proposed framework is effective in restoring the fidelity of the closed-loop integration. This improvement is showcased through improved performance and transparency, which leads to excellent command-tracking performance.

Spiking Neural Networks (SNNs) offer a biologically inspired approach to computer vision that can lead to more efficient processing of visual data with reduced energy consumption. However, maintaining homeostasis within these networks is challenging, as it requires continuous adjustment of neural responses to preserve equilibrium and optimal processing efficiency amidst diverse and often unpredictable input signals. In response to these challenges, we propose the Asynchronous Bioplausible Neuron (ABN), a dynamic spike firing mechanism to auto-adjust the variations in the input signal. Comprehensive evaluation across various datasets demonstrates ABN's enhanced performance in image classification and segmentation, maintenance of neural equilibrium, and energy efficiency., Comment: 10 pages

Reinforcement Learning (RL) is an emerging approach to control many dynamical systems for which classical control approaches are not applicable or insufficient. However, the resultant policies may not generalize to variations in the parameters that the system may exhibit. This paper presents a powerful yet simple algorithm in which collaboration is facilitated between RL agents that are trained independently to perform the same task but with different system parameters. The independency among agents allows the exploitation of multi-core processing to perform parallel training. Two examples are provided to demonstrate the effectiveness of the proposed technique. The main demonstration is performed on a quadrotor with slung load tracking problem in a real-time experimental setup. It is shown that integrating the developed algorithm outperforms individual policies by reducing the RMSE tracking error. The robustness of the ensemble is also verified against wind disturbance., Comment: arXiv admin note: text overlap with arXiv:2311.05013

Reinforcement learning is an emerging approach to control dynamical systems for which classical approaches are difficult to apply. However, trained agents may not generalize against the variations of system parameters. This paper presents the concept of dual-scale homogeneity, an important property in understating the scaling behavior of nonlinear systems. Furthermore, it also presents an effective yet simple approach to designing a parameter-dependent control law that homogenizes a nonlinear system. The presented approach is applied to two systems, demonstrating its ability to provide a consistent performance irrespective of parameters variations. To demonstrate the practicality of the proposed approach, the control policy is generated by a deep deterministic policy gradient to control the load position of a quadrotor with a slung load. The proposed synergy between the homogeneity transformations and reinforcement learning yields superior performance compared to other recent learning-based control techniques. It achieves a success rate of 96% in bringing the load to its designated target with a 3D RMSE of 0.0253 m. The video that shows the experimental results along with a summary of the paper is available at this link.

A thorough analysis of wheel-terrain interaction is critical to ensure the safe and efficient operation of space rovers on extraterrestrial surfaces like the Moon or Mars. This paper presents an approach for developing and experimentally validating a virtual wheel-terrain interaction model for the UAE Rashid rover. The model aims to improve the fidelity and capability of current simulation methods for space rovers and facilitate the design, evaluation, and control of their locomotion systems. The proposed method considers various factors, such as wheel grouser properties, wheel slippage, loose soil properties, and interaction mechanics. The model accuracy was validated through experiments on a Test-rig testbed that simulated lunar soil conditions. In specific, a set of experiments was carried out to test the behaviors acted on a Grouser-Rashid rover wheel by the lunar soil with different slip ratios of 0, 0.25, 0.50, and 0.75. The obtained results demonstrate that the proposed simulation method provides a more accurate and realistic simulation of the wheel-terrain interaction behavior and provides insight into the overall performance of the rover, Comment: Submitted to ICAR2023 conference

Event cameras triggered a paradigm shift in the computer vision community delineated by their asynchronous nature, low latency, and high dynamic range. Calibration of event cameras is always essential to account for the sensor intrinsic parameters and for 3D perception. However, conventional image-based calibration techniques are not applicable due to the asynchronous, binary output of the sensor. The current standard for calibrating event cameras relies on either blinking patterns or event-based image reconstruction algorithms. These approaches are difficult to deploy in factory settings and are affected by noise and artifacts degrading the calibration performance. To bridge these limitations, we present E-Calib, a novel, fast, robust, and accurate calibration toolbox for event cameras utilizing the asymmetric circle grid, for its robustness to out-of-focus scenes. The proposed method is tested in a variety of rigorous experiments for different event camera models, on circle grids with different geometric properties, and under challenging illumination conditions. The results show that our approach outperforms the state-of-the-art in detection success rate, reprojection error, and estimation accuracy of extrinsic parameters., Comment: 13 pages, 6 tables, 15 figures

In the context of robotic grasping, object segmentation encounters several difficulties when faced with dynamic conditions such as real-time operation, occlusion, low lighting, motion blur, and object size variability. In response to these challenges, we propose the Graph Mixer Neural Network that includes a novel collaborative contextual mixing layer, applied to 3D event graphs formed on asynchronous events. The proposed layer is designed to spread spatiotemporal correlation within an event graph at four nearest neighbor levels parallelly. We evaluate the effectiveness of our proposed method on the Event-based Segmentation (ESD) Dataset, which includes five unique image degradation challenges, including occlusion, blur, brightness, trajectory, scale variance, and segmentation of known and unknown objects. The results show that our proposed approach outperforms state-of-the-art methods in terms of mean intersection over the union and pixel accuracy. Code available at: https://github.com/sanket0707/GNN-Mixer.git, Comment: 9 pages, 6 figures

Countersink inspection is crucial in various automated assembly lines, especially in the aerospace and automotive sectors. Advancements in machine vision introduced automated robotic inspection of countersinks using laser scanners and monocular cameras. Nevertheless, the aforementioned sensing pipelines require the robot to pause on each hole for inspection due to high latency and measurement uncertainties with motion, leading to prolonged execution times of the inspection task. The neuromorphic vision sensor, on the other hand, has the potential to expedite the countersink inspection process, but the unorthodox output of the neuromorphic technology prohibits utilizing traditional image processing techniques. Therefore, novel event-based perception algorithms need to be introduced. We propose a countersink detection approach on the basis of event-based motion compensation and the mean-shift clustering principle. In addition, our framework presents a robust event-based circle detection algorithm to precisely estimate the depth of the countersink specimens. The proposed approach expedites the inspection process by a factor of 10$\times$ compared to conventional countersink inspection methods. The work in this paper was validated for over 50 trials on three countersink workpiece variants. The experimental results show that our method provides a precision of 0.025 mm for countersink depth inspection despite the low resolution of commercially available neuromorphic cameras., Comment: 14 pages, 11 figures, 7 tables, submitted to Journal of Intelligent Manufacturing

In this paper, we present a real-time system identification method based on relay feedback testing with applications to multirotor unmanned aerial vehicles. The proposed identification method provides an alternative to the expensive lab testing of certain UAV dynamic parameters. Moreover, it has the advantage of identifying the parameters that get changed throughout the operation of the UAV, which requires onboard identification methods. The modified relay feedback test (MRFT) is used to generate stable limit cycles at frequency points that reveal the underlying UAV dynamics. The locus of the perturbed relay system (LPRS) is used to predict the exact amplitude and frequency of these limit cycles. Real-time identification is achieved by using the homogeneity properties of the MRFT and the LPRS which are proven in this paper. The proposed identification method was tested experimentally to estimate the aerodynamic parameters as well as the onboard sensor's time delay parameters. The MRFT testing takes a few seconds to perform, and the identification computations take an average of 0.2 seconds to complete in modern embedded computers. The proposed identification method is compared against state-of-the-art alternatives. Advantages in identification accuracy and quantification of uncertainty in estimated parameters are shown., Comment: 9 pages, 5 figures, This work has been submitted to the IEEE for possible publication

Object segmentation for robotic grasping under dynamic conditions often faces challenges such as occlusion, low light conditions, motion blur and object size variance. To address these challenges, we propose a Deep Learning network that fuses two types of visual signals, event-based data and RGB frame data. The proposed Bimodal SegNet network has two distinct encoders, one for each signal input and a spatial pyramidal pooling with atrous convolutions. Encoders capture rich contextual information by pooling the concatenated features at different resolutions while the decoder obtains sharp object boundaries. The evaluation of the proposed method undertakes five unique image degradation challenges including occlusion, blur, brightness, trajectory and scale variance on the Event-based Segmentation (ESD) Dataset. The evaluation results show a 6-10\% segmentation accuracy improvement over state-of-the-art methods in terms of mean intersection over the union and pixel accuracy. The model code is available at https://github.com/sanket0707/Bimodal-SegNet.git, Comment: 8 Pages

Taking advantage of an event-based camera, the issues of motion blur, low dynamic range and low time sampling of standard cameras can all be addressed. However, there is a lack of event-based datasets dedicated to the benchmarking of segmentation algorithms, especially those that provide depth information which is critical for segmentation in occluded scenes. This paper proposes a new Event-based Segmentation Dataset (ESD), a high-quality 3D spatial and temporal dataset for object segmentation in an indoor cluttered environment. Our proposed dataset ESD comprises 145 sequences with 14,166 RGB frames that are manually annotated with instance masks. Overall 21.88 million and 20.80 million events from two event-based cameras in a stereo-graphic configuration are collected, respectively. To the best of our knowledge, this densely annotated and 3D spatial-temporal event-based segmentation benchmark of tabletop objects is the first of its kind. By releasing ESD, we expect to provide the community with a challenging segmentation benchmark with high quality.

This paper analyzes the simulation to reality gap in reinforcement learning (RL) cyber-physical systems with fractional delays (i.e. delays that are non-integer multiple of the sampling period). The consideration of fractional delay has important implications on the nature of the cyber-physical system considered. Systems with delays are non-Markovian, and the system state vector needs to be extended to make the system Markovian. We show that this is not possible when the delay is in the output, and the problem would always be non-Markovian. Based on this analysis, a sampling scheme is proposed that results in efficient RL training and agents that perform well in realistic multirotor unmanned aerial vehicle simulations. We demonstrate that the resultant agents do not produce excessive oscillations, which is not the case with RL agents that do not consider time delay in the model., Comment: 6 pages,4 figures, Submitted to ICRA2023

In this paper, we analyze the effect of time delay dynamics on controller design for Unmanned Aerial Vehicles (UAVs) with vision based navigation. Time delay is an inevitable phenomenon in cyber-physical systems, and has important implications on controller design and trajectory generation for UAVs. The impact of time delay on UAV dynamics increases with the use of the slower vision based navigation stack. We show that the existing models in the literature, which exclude time delay, are unsuitable for controller tuning since a trivial solution for minimizing an error cost functional always exists. The trivial solution that we identify suggests use of infinite controller gains to achieve optimal performance, which contradicts practical findings. We avoid such shortcomings by introducing a novel nonlinear time delay model for UAVs, and then obtain a set of linear decoupled models corresponding to each of the UAV control loops. The cost functional of the linearized time delay model of angular and altitude dynamics is analyzed, and in contrast to the delay-free models, we show the existence of finite optimal controller parameters. Due to the use of time delay models, we experimentally show that the proposed model accurately represents system stability limits. Due to time delay consideration, we achieved a tracking results of RMSE 5.01 cm when tracking a lemniscate trajectory with a peak velocity of 2.09 m/s using visual odometry (VO) based UAV navigation, which is on par with the state-of-the-art.

This paper presents a theoretical study on the ability of multi-rotor aerial vehicles (MRAVs) with tiltable propellers to achieve and sustain static hovering at different orientations. To analyze the ability of MRAVs with tiltable propellers to achieve static hovering, a novel linear map between the platform's control inputs and applied forces and moments is introduced. The relation between the introduced map and the platform's ability to hover at different orientations is developed. Correspondingly, the conditions for MRAVs with tiltable propellers to realize and sustain static hovering are detailed. A numerical metric is then introduced, which reflects the ability of MRAVs to sustain static hovering at different orientations. A subclass of MRAVs with tiltable propellers is defined as the Critically Statically Hoverable platforms (CSH), where CSH platforms are MRAVs that cannot sustain static hovering with fixed propellers, but can achieve static hovering with tilting propellers. Finally, extensive simulations are conducted to test and validate the above findings, and to demonstrate the effect of the proposed numerical metric on the platform's dynamics.

The field of Unmanned Aerial Vehicles (UAVs) has reached a high level of maturity in the last few years. Hence, bringing such platforms from closed labs, to day-to-day interactions with humans is important for commercialization of UAVs. One particular human-UAV scenario of interest for this paper is the payload handover scheme, where a UAV hands over a payload to a human upon their request. In this scope, this paper presents a novel real-time human-UAV interaction detection approach, where Long short-term memory (LSTM) based neural network is developed to detect state profiles resulting from human interaction dynamics. A novel data pre-processing technique is presented; this technique leverages estimated process parameters of training and testing UAVs to build dynamics invariant testing data. The proposed detection algorithm is lightweight and thus can be deployed in real-time using off the shelf UAV platforms; in addition, it depends solely on inertial and position measurements present on any classical UAV platform. The proposed approach is demonstrated on a payload handover task between multirotor UAVs and humans. Training and testing data were collected using real-time experiments. The detection approach has achieved an accuracy of 96\%, giving no false positives even in the presence of external wind disturbances, and when deployed and tested on two different UAVs., Comment: 13 pages, 13 figures, submitted to IEEE access, A supplementary video for the work presented in this paper can be accessed from https://youtu.be/29N_OXBl1mc

Space exploration has witnessed revolutionary changes upon landing of the Perseverance Rover on the Martian surface and demonstrating the first flight beyond Earth by the Mars helicopter, Ingenuity. During their mission on Mars, Perseverance Rover and Ingenuity collaboratively explore the Martian surface, where Ingenuity scouts terrain information for rover's safe traversability. Hence, determining the relative poses between both the platforms is of paramount importance for the success of this mission. Driven by this necessity, this work proposes a robust relative localization system based on a fusion of neuromorphic vision-based measurements (NVBMs) and inertial measurements. The emergence of neuromorphic vision triggered a paradigm shift in the computer vision community, due to its unique working principle delineated with asynchronous events triggered by variations of light intensities occurring in the scene. This implies that observations cannot be acquired in static scenes due to illumination invariance. To circumvent this limitation, high frequency active landmarks are inserted in the scene to guarantee consistent event firing. These landmarks are adopted as salient features to facilitate relative localization. A novel event-based landmark identification algorithm using Gaussian Mixture Models (GMM) is developed for matching the landmarks correspondences formulating our NVBMs. The NVBMs are fused with inertial measurements in proposed state estimators, landmark tracking Kalman filter (LTKF) and translation decoupled Kalman filter (TDKF) for landmark tracking and relative localization, respectively. The proposed system was tested in a variety of experiments and has outperformed state-of-the-art approaches in accuracy and range.

The manufacturing industry is currently witnessing a paradigm shift with the unprecedented adoption of industrial robots, and machine vision is a key perception technology that enables these robots to perform precise operations in unstructured environments. However, the sensitivity of conventional vision sensors to lighting conditions and high-speed motion sets a limitation on the reliability and work-rate of production lines. Neuromorphic vision is a recent technology with the potential to address the challenges of conventional vision with its high temporal resolution, low latency, and wide dynamic range. In this paper and for the first time, we propose a novel neuromorphic vision based controller for faster and more reliable machining operations, and present a complete robotic system capable of performing drilling tasks with sub-millimeter accuracy. Our proposed system localizes the target workpiece in 3D using two perception stages that we developed specifically for the asynchronous output of neuromorphic cameras. The first stage performs multi-view reconstruction for an initial estimate of the workpiece's pose, and the second stage refines this estimate for a local region of the workpiece using circular hole detection. The robot then precisely positions the drilling end-effector and drills the target holes on the workpiece using a combined position-based and image-based visual servoing approach. The proposed solution is validated experimentally for drilling nutplate holes on workpieces placed arbitrarily in an unstructured environment with uncontrolled lighting. Experimental results prove the effectiveness of our solution with an average positional errors of less than 0.1 mm, and demonstrate that the use of neuromorphic vision overcomes the lighting and speed limitations of conventional cameras., Comment: 14 pages, 11 figures, 2 tables, submitted to Robotics and Computer-Integrated Manufacturing

Neuromorphic vision is a bio-inspired technology that has triggered a paradigm shift in the computer-vision community and is serving as a key-enabler for a multitude of applications. This technology has offered significant advantages including reduced power consumption, reduced processing needs, and communication speed-ups. However, neuromorphic cameras suffer from significant amounts of measurement noise. This noise deteriorates the performance of neuromorphic event-based perception and navigation algorithms. In this paper, we propose a novel noise filtration algorithm to eliminate events which do not represent real log-intensity variations in the observed scene. We employ a Graph Neural Network (GNN)-driven transformer algorithm, called GNN-Transformer, to classify every active event pixel in the raw stream into real-log intensity variation or noise. Within the GNN, a message-passing framework, called EventConv, is carried out to reflect the spatiotemporal correlation among the events, while preserving their asynchronous nature. We also introduce the Known-object Ground-Truth Labeling (KoGTL) approach for generating approximate ground truth labels of event streams under various illumination conditions. KoGTL is used to generate labeled datasets, from experiments recorded in chalenging lighting conditions. These datasets are used to train and extensively test our proposed algorithm. When tested on unseen datasets, the proposed algorithm outperforms existing methods by 8.8% in terms of filtration accuracy. Additional tests are also conducted on publicly available datasets to demonstrate the generalization capabilities of the proposed algorithm in the presence of illumination variations and different motion dynamics. Compared to existing solutions, qualitative results verified the superior capability of the proposed algorithm to eliminate noise while preserving meaningful scene events.

Robotic vision plays a key role for perceiving the environment in grasping applications. However, the conventional framed-based robotic vision, suffering from motion blur and low sampling rate, may not meet the automation needs of evolving industrial requirements. This paper, for the first time, proposes an event-based robotic grasping framework for multiple known and unknown objects in a cluttered scene. Compared with standard frame-based vision, neuromorphic vision has advantages of microsecond-level sampling rate and no motion blur. Building on that, the model-based and model-free approaches are developed for known and unknown objects' grasping respectively. For the model-based approach, event-based multi-view approach is used to localize the objects in the scene, and then point cloud processing allows for the clustering and registering of objects. Differently, the proposed model-free approach utilizes the developed event-based object segmentation, visual servoing and grasp planning to localize, align to, and grasp the targeting object. The proposed approaches are experimentally validated with objects of different sizes, using a UR10 robot with an eye-in-hand neuromorphic camera and a Barrett hand gripper. Moreover, the robustness of the two proposed event-based grasping approaches are validated in a low-light environment. This low-light operating ability shows a great advantage over the grasping using the standard frame-based vision. Furthermore, the developed model-free approach demonstrates the advantage of dealing with unknown object without prior knowledge compared to the proposed model-based approach., Comment: 37 pages

Vision based control of Unmanned Aerial Vehicles (UAVs) has been adopted by a wide range of applications due to the availability of low-cost on-board sensors and computers. Tuning such systems to work properly requires extensive domain specific experience, which limits the growth of emerging applications. Moreover, obtaining performance limits of UAV based visual servoing is difficult due to the complexity of the models used. In this paper, we propose a novel noise tolerant approach for real-time identification and tuning of visual servoing systems, based on deep neural networks (DNN) classification of system response generated by the modified relay feedback test (MRFT). The proposed method, called DNN with noise protected MRFT (DNN-NP-MRFT), can be used with a multitude of vision sensors and estimation algorithms despite the high levels of sensor's noise. Response of DNN-NP-MRFT to noise perturbations is investigated and its effect on identification and tuning performance is analyzed. The proposed DNN-NP-MRFT is able to detect performance changes due to the use of high latency vision sensors, or due to the integration of inertial measurement unit (IMU) measurements in the UAV states estimation. Experimental identification closely matches simulation results, which can be used to explain system behaviour and predict the closed loop performance limits for a given hardware and software setup. We also demonstrate the ability of DNN-NP-MRFT tuned UAVs to reject external disturbances like wind, or human push and pull. Finally, we discuss the advantages of the proposed DNN-NP-MRFT visual servoing design approach compared with other approaches in literature.

Control performance of Unmanned Aerial Vehicles (UAVs) is directly affected by their ability to estimate their states accurately. With the increasing popularity of autonomous UAV solutions in real world applications, it is imperative to develop robust adaptive estimators that can ameliorate sensor noises in low-cost UAVs. Utilizing the knowledge of UAV dynamics in estimation can provide significant advantages, but remains challenging due to the complex and expensive pre-flight experiments required to obtain UAV dynamic parameters. In this paper, we propose two decoupled dynamic model based Extended Kalman Filters for UAVs, that provide high rate estimates for position, and velocity of rotational and translational states, as well as filtered inertial acceleration. The dynamic model parameters are estimated online using the Deep Neural Network and Modified Relay Feedback Test (DNN-MRFT) framework, without requiring any prior knowledge of the UAV physical parameters. The designed filters with real-time identified process model parameters are tested experimentally and showed two advantages. Firstly, smooth and lag-free estimates of the UAV rotational speed and inertial acceleration are obtained, and used to improve the closed loop system performance, reducing the controller action by over 6 %. Secondly, the proposed approach enabled the UAV to track aggressive trajectories with low rate position measurements, a task usually infeasible under those conditions. The experimental data shows that we achieved estimation performance matching other methods that requires full knowledge of the UAV parameters.

The demand for accurate and fast trajectory tracking for multirotor Unmanned Aerial Vehicles (UAVs) have grown recently due to advances in UAV avionics technology and application domains. In many applications, the multirotor UAV is required to accurately perform aggressive maneuvers in challenging scenarios like the presence of external wind disturbances or in-flight payload changes. In this paper, we propose a systematic controller tuning approach based on identification results obtained by a recently developed Deep Neural Networks with the Modified Relay Feedback Test (DNN-MRFT) algorithm. We formulate a linear equivalent representation suitable for DNN-MRFT using feedback linearization. This representation enables the analytical investigation of different controller structures and tuning settings, and captures the non-linearity trends of the system. With this approach, the trade-off between performance and robustness in design was made possible which is convenient for the design of controllers of UAVs operating in uncertain environments. We demonstrate that our approach is adaptive and robust through a set of experiments, where accurate trajectory tracking is maintained despite significant changes to the UAV aerodynamic characteristics and the application of wind disturbance. Due to the model-based system design, it was possible to obtain low discrepancy between simulation and experimental results which is beneficial for potential use of the proposed approach for real-time model-based planning and fault detection tasks. We obtained RMSE of $3.59 \; cm$ when tracking aggressive trajectories in the presence of strong wind, which is on par with state-of-the-art.

Low cost real-time identification of multirotor unmanned aerial vehicle (UAV) dynamics is an active area of research supported by the surge in demand and emerging application domains. Such real-time identification capabilities shorten development time and cost, making UAVs' technology more accessible, and enable a wide variety of advanced applications. In this paper, we present a novel comprehensive approach, called DNN-MRFT, for real-time identification and tuning of multirotor UAVs using the Modified Relay Feedback Test (MRFT) and Deep Neural Networks (DNN). The main contribution is the development of a generalized framework for the application of DNN-MRFT to higher-order systems. One of the notable advantages of DNN-MRFT is the exact estimation of identified process gain, which mitigates the inaccuracies introduced due to the use of the describing function method in approximating the response of Lure's systems. A secondary contribution is a generalized controller based on DNN-MRFT that takes-off a UAV with unknown dynamics and identifies the inner loops dynamics in-flight. Using the developed framework, DNN-MRFT is sequentially applied to the outer translational loops of the UAV utilizing in-flight results obtained for the inner attitude loops. DNN-MRFT takes on average 15 seconds to get the full knowledge of multirotor UAV dynamics and without any further tuning or calibration the UAV would be able to pass through a vertical window, and accurately follow trajectories achieving state-of-the-art performance. Such demonstrated accuracy, speed, and robustness of identification pushes the limits of state-of-the-art in real-time identification of UAVs., Comment: 19 pages, 11 figures, Submitted to IEEE Transactions on Control System Technology. A supplementary video for the work presented in this paper can be accessed from https://youtu.be/07RtnZxTJRM

This paper proposes a novel parametric identification approach for linear systems using Deep Learning (DL) and the Modified Relay Feedback Test (MRFT). The proposed methodology utilizes MRFT to reveal distinguishing frequencies about an unknown process; which are then passed to a trained DL model to identify the underlying process parameters. The presented approach guarantees stability and performance in the identification and control phases respectively, and requires few seconds of observation data to infer the dynamic system parameters. Quadrotor Unmanned Aerial Vehicle (UAV) attitude and altitude dynamics were used in simulation and experimentation to verify the presented methodology. Results show the effectiveness and real-time capabilities of the proposed approach, which outperforms the conventional Prediction Error Method in terms of accuracy, robustness to biases, computational efficiency and data requirements., Comment: 13 pages, 9 figures. Submitted to IEEE access. A supplementary video for the work presented in this paper can be accessed at: https://www.youtube.com/watch?v=dz3WTFU7W7c. This version includes minor style edits for appendix and references

Slip detection is essential for robots to make robust grasping and fine manipulation. In this paper, a novel dynamic vision-based finger system for slip detection and suppression is proposed. We also present a baseline and feature based approach to detect object slips under illumination and vibration uncertainty. A threshold method is devised to autonomously sample noise in real-time to improve slip detection. Moreover, a fuzzy based suppression strategy using incipient slip feedback is proposed for regulating the grip force. A comprehensive experimental study of our proposed approaches under uncertainty and system for high-performance precision manipulation are presented. We also propose a slip metric to evaluate such performance quantitatively. Results indicate that the system can effectively detect incipient slip events at a sampling rate of 2kHz ($\Delta t = 500\mu s$) and suppress them before a gross slip occurs. The event-based approach holds promises to high precision manipulation task requirement in industrial manufacturing and household services., Comment: 18 pages, 14 figures

Robotic vision plays a major role in factory automation to service robot applications. However, the traditional use of frame-based camera sets a limitation on continuous visual feedback due to their low sampling rate and redundant data in real-time image processing, especially in the case of high-speed tasks. Event cameras give human-like vision capabilities such as observing the dynamic changes asynchronously at a high temporal resolution ($1\mu s$) with low latency and wide dynamic range. In this paper, we present a visual servoing method using an event camera and a switching control strategy to explore, reach and grasp to achieve a manipulation task. We devise three surface layers of active events to directly process stream of events from relative motion. A purely event based approach is adopted to extract corner features, localize them robustly using heat maps and generate virtual features for tracking and alignment. Based on the visual feedback, the motion of the robot is controlled to make the temporal upcoming event features converge to the desired event in spatio-temporal space. The controller switches its strategy based on the sequence of operation to establish a stable grasp. The event based visual servoing (EVBS) method is validated experimentally using a commercial robot manipulator in an eye-in-hand configuration. Experiments prove the effectiveness of the EBVS method to track and grasp objects of different shapes without the need for re-tuning., Comment: 8 pages, 10 figures