Search

My dissertation focuses on three research problems to investigate how the robot's behavior leads to a qualitative and quantitative explanation of neural activities, and vice versa, that is, how neural activities lead to behavior. In the first problem, we simulated a rat in a robot simulator to replicate the behavior and neural activity observed in rats during a spatial and working memory task. A recurrent neural network (RNN) with sensory and vision inputs was evolved to control the robot motor wheels and navigate a virtual T-maze. Our current findings suggest that neurons in the RNN are performing mixed selectivity and conjunctive coding. Moreover, the RNN activity resembles spatial information and trajectory-dependent coding observed in the hippocampus. In the second problem, we developed a goal-driven perception algorithm inspired by effects of the cholinergic (ACh) and noradrenergic (NE) neuromodulatory systems on attention and tracking uncertainties. We tested the network architecture, which extended the contrastive excitation backprop (c-EB), in a noisy MNIST-pair task and an action-based human support robot task. The network architecture could quickly learn the context without supervision, flexibly apply attention to the appropriate goal, and rapidly detect and re-adapt to context changes. In the third problem, we developed a reservoir-based spiking neural network (r-SNN) to classify three terrain types in a botanical garden. The input spike trains were generated from the linear accelerometer, gyroscope, and image data collected by a six-wheel Android-based robot (ABR). Our r-SNN terrain prediction can be used to evaluate the cost of traversal for path planning. It is a promising approach to develop a complete neuromorphic robot navigation system capable of operating over long durations with minimal power consumption. We suggest that neurorobotic investigation of biologically plausible neural networks can be a powerful methodology for understanding neuroscience, as well as for artificial intelligence and machine learning.

Legged robot navigation in extreme environments can hinder the use of cameras and laser scanners due to darkness, air obfuscation or sensor damage. In these conditions, proprioceptive sensing will continue to work reliably. In this paper, we propose a purely proprioceptive localization algorithm which fuses information from both geometry and terrain class, to localize a legged robot within a prior map. First, a terrain classifier computes the probability that a foot has stepped on a particular terrain class from sensed foot forces. Then, a Monte Carlo-based estimator fuses this terrain class probability with the geometric information of the foot contact points. Results are demonstrated showing this approach operating online and onboard a ANYmal B300 quadruped robot traversing a series of terrain courses with different geometries and terrain types over more than 1.2km. The method keeps the localization error below 20cm using only the information coming from the feet, IMU, and joints of the quadruped., Autonomous Robots. arXiv admin note: substantial text overlap with arXiv:2005.01567

In this paper, a hybrid method based on deep learning is proposed to visually classify terrains encountered by mobile robots. Considering the limited computing resource on mobile robots and the requirement for high classification accuracy, the proposed hybrid method combines a convolutional neural network with a support vector machine to keep a high classification accuracy while improve work efficiency. The key idea is that the convolutional neural network is used to finish a multi-class classification and simultaneously the support vector machine is used to make a two-class classification. The two-class classification performed by the support vector machine is aimed at one kind of terrain that users are mostly concerned with. Results of the two classifications will be consolidated to get the final classification result. The convolutional neural network used in this method is modified for the on-board usage of mobile robots. In order to enhance efficiency, the convolutional neural network has a simple architecture. The convolutional neural network and the support vector machine are trained and tested by using RGB images of six kinds of common terrains. Experimental results demonstrate that this method can help robots classify terrains accurately and efficiently. Therefore, the proposed method has a significant potential for being applied to the on-board usage of mobile robots.

The use of Bandt–Pompe probability distributions and descriptors of information theory has been presenting satisfactory results with low computational cost in the time series analysis literature [1]–[3]. However, these tools have limitations when applied to data without time dependency. Given this context, we present a newly proposed technique for texture analysis and classification based on the Bandt–Pompe symbolization for SAR data. It consists of linearizing a 2-D patch of the image using the Hilbert–Peano curve, build an ordinal pattern transition graph that considers the data amplitude encoded into the weight of the edges, obtain a probability distribution function derived from this graph, and compute information theory descriptors (permutation entropy and statistical complexity) from this distribution and use them as features to feed a classifier. The ordinal pattern graph we propose considers that the edges’ weight is related to the absolute difference of observations, which encodes the information about the data amplitude. This modification considers the unfavorable signal-to-noise ratio of SAR images and leads to the characterization of several types of textures. Experiments with data from Munich urban areas, Guatemala forest regions, and Cape Canaveral ocean samples show the effectiveness of our technique in homogeneous areas, achieving satisfactory separability levels. The two descriptors chosen in this work are easy and quick to calculate and are used as input for a k-nearest neighbor classifier. Experiments show that this technique presents results similar to state-of-the-art techniques that employ a much larger number of features and, consequently, impose a higher computational cost.

Since the existing terrain classification algorithm based on deep learning is not ideal for unbalanced PolSAR classification, a effective terrain classification algorithm based on wavelet kernel sparse deep coding network under unbalanced data set is proposed in this paper. The algorithm firstly adopts a structured sparse operation so as to enhance the accuracy of feature propagation and reduce the amount of stored data, where the unimportant parameter connections in each group are gradually reduced by dividing the network convolution kernel into multiple groups during the training process; The wavelet kernel-based classifier is used instead of the Sigmoid function to classify and identify features for different terrain, which has high generalization performance for small sample, nonlinear and high-dimensional mode classification problems. The experimental results show that our proposed classification algorithm can improve the classification performance of unbalanced samples, and improve the classification efficiency while ensuring the accuracy of classification.

Decision tree method has been applied to POLSAR image classification, due to its capability to interpret the scattering characteristics as well as good classification accuracy. Compared with popular machine learning classifiers, decision tree approach can explain the scattering process of certain type of targets by use of the polarimetric features at the tree nodes. Except the interpretability, decision tree approach could be transplanted to other data set without training process for the same terrain types, since the polarimetric features are inherently connected to the physical scattering properties. Currently, decision tree based classifiers, typically employ one single polarimetric feature at the nodes of the tree. The idea to increase the number of the polarization features at the decision tree node is expected to improve the classification result, which combine two or more polarimetric features to form a two or high dimension feature space. In this way, the classes which cannot be discriminated with one feature could possibly be separated with the space constructed by several features. However, it also inevitably leads to an increase in the computational burden. In fact, not all nodes require very high-dimensional feature space to achieve high classification precision. Therefore, in this article we proposed that the dimension of the feature space used in the decision tree nodes is adaptively changed from one to three, due to the separability of the classes under this node. The developed classification method is examined by the classical AIRSAR data in Flevoland area of the Netherlands, as well as GaoFen-3 data in Hulunbuir of China. The experiments show that the classification performance is superior to the fixed dimension feature decision tree methods, with less and reasonable computation time. Besides, the transferability of polarimetric features obtained by decision tree is preliminarily demonstrated in the application to another AIRSAR data.

Terrain classification is an important topic in polarimetric synthetic aperture radar (PolSAR) image processing and interpretation. A novel PolSAR classification method based on the three-channel convolutional neural network (Tc-CNN) is proposed and this method can effectively take the advantage of unlabeled samples to improve the performance of classification with a small number of labeled samples. Several strategies are included in the proposed method. First, in order to take the advantage of unlabeled samples, a data enhancement method based on the neighborhood nearest neighbor propagation method is proposed to enlarge the number of labeled samples. Second, to increase the role of central pixel in convolutional neural network classification based on the pixel, a spatial weighted method is proposed to increase the weight of central pixel features and weak the weight of other types of pixel features. Third, a specific deep model for PolSAR image classification (named Tc-CNN) is proposed, which can obtain more scale and deep polarization information to improve the classification results. The experimental results show that the proposed method achieves a much better performance than the existing classification methods when the number of labeled samples is few.

Amongst the utilizations of Polarimetric Synthetic Aperture Radar (PoISAR) data, semi-supervised terrain classification is much in demand. Samples of the same category are distributed in multiple regions of a PolSAR image, resulting in differences in the feature distributions of samples of the same category located in different regions. In addition, some samples of confusable categories, and samples of different categories located near the edges, have small feature differences in a PolSAR image. To address this problem, we introduce a graph regular loss constructed from pseudo-labels to constrain the intra-class similarity and inter-class similarity of features, and improve the discriminative property of features. In addition, since the quality of pseudo-labels affects the optimization of the model by the graph regular loss, we introduce the idea of clustering into the teacher-student model to improve the quality of pseudo-labels. Experiments on real PolSAR data show that our proposed method achieves an excellent performance.

To achieve effective deep fusion features for improving the classification accuracy of hyperspectral remote sensing images (HRSIs), a pixel frequency spectrum feature is presented and introduced to convolutional neural networks (CNNs). Firstly, the fast Fourier transform is performed on each spectral pixel to obtain the amplitude spectrum, i.e., the pixel frequency spectrum feature. Then, the obtained pixel frequency spectrum is combined with the spectral pixel to form a mixed feature, i.e., spectral and frequency spectrum mixed feature (SFMF). Several multi-branch CNNs fed with pixel frequency spectrum, SFMF, spectral pixel, and spatial features are designed for extracting deep fusion features. A pre-learning strategy, i.e., basic single branch CNNs are used to pre-learn the weights of a multi-branch CNN, is also presented for improving the network convergence speed and avoiding the network from getting into a locally optimal solution to a certain extent. And after reducing the dimensionality of SFMF by principal component analysis (PCA), a 3-dimensionality (3-D) CNN is also designed to further extract the joint spatial-SFMF feature. The experimental results of three real HRSIs show that adding the presented frequency spectrum feature into CNNs can achieve better recognition results, which in turn proves that the presented multi-branch CNNs can obtain the deep fusion features with more discriminant information.

Wheel-terrain interaction plays a critical role for vehicle mobility on natural terrain, such as in agricultural, planetary exploration and off-road settings. Estimation of the terrain characteristics and the way they affect traversability is essential for the vehicle to better plan its safest and energy-efficient path. This work proposes a novel approach to learn and predict from a distance the motion resistance encountered by a robotic vehicle, while traversing natural soil, by using visual information from a stereovision device. To this end, terrain appearance and geometry information are first correlated to resistance torque measurements during a learning phase via two alternative regression approaches, namely Least-Squares Boosting and Long-Short Term Memory Recurrent Neural Network. Then, such a relationship is exploited to predict motion resistance remotely, based on visual data only. Results obtained in preliminary experimental tests on ploughed and compact terrain are presented to show the feasibility of the proposed method.

In practical applications of automated terrain classification from high-resolution polarimetric synthetic aperture radar (PolSAR) imagery, different terrain types may inherently contain a high level of internal variability, as when a broadly defined class (e.g., “trees”) contains elements arising from multiple subclasses (pine, oak, and willow). In addition, real-world factors such as the time of year of a collection, the moisture content of the scene, the imaging geometry, and the radar system parameters can all increase the variability observed within each class. Such variability challenges the ability of classifiers to maintain a high level of sensitivity in recognizing diverse elements that are within-class, without sacrificing their selectivity in rejecting out-of-class elements. In an effort to gauge the degree to which classifiers respond robustly in the presence of intraclass variability and generalize to untrained scenes and conditions, we compare the performance of a suite of classifiers across six broad terrain categories from a large set of polarimetric synthetic aperture radar (PolSAR) image sets. The main contributions of this article are as follows: 1) an analysis of the robustness of a variety of current state-of-the art classification algorithms to intraclass variability found in PolSAR image sets, and 2) the associated PolSAR image and feature data that Sandia is releasing to the research community with this publication. The analysis of the classification algorithms we provide will serve as a benchmark of performance for the future PolSAR terrain classification algorithm research and development enabled by the image sets and data provided. By sharing our analysis and high-resolution fully polarimetric Sandia data with the research community, we enable others to develop and assess a new generation of robust terrain classification algorithms for PolSAR.

The use of electric vehicles (EVs) has been viewed as a potential solution to reduce the oil dependence in transportation. Nevertheless, the performance of the EV changes according to the nature and characteristics of the terrain. This work focuses in modelling the instantaneous power consumption (IPC) of an EV as it traverses through different types of terrains, classified using an artificial vision system. The EV is sensorized with voltage and current sensors, accurate real time kinematics device, an artificial vision system and a real time processing unit, to perform field tests. To see the advantages of using terrain information for energy prognosis in EVs, we compared our proposal with the information provided by the manufacturer and previously published approaches. Our system showed to be up to 95% more accurate when tested over clay, gravel, pavement and grass in both short term and long term trials, and it does not need of a complete knowledge of the system’s model.

Extraterrestrial celestial patrol missions have introduced very strict requirements for the performance of rovers, due to their high cost. Vision-based or Lidar-based environment sensing technology has matured. However, due to its perceptual characteristics, it is impossible to predict the traversability of the terrain completely, and it lacks the judgment of the physical properties of the terrain, such as the degree of hardness and softness. Due to the spectrum of risks that the rover is facing, a wide range of detection processes is required. This research paper proposes a terrain classification approach based on 3-D vibrations induced in the rover structure by the wheel-terrain interaction. Initially, the acceleration information of the three directions is obtained by using the Inertial measurement unit of the rover. Then, the characteristics of the vibrations of the known terrain are learned. The Fast Fourier Transformation (FFT) is used to transform the labeled three-axis vibration vectors into a frequency domain. Then the training feature vectors are obtained through normalization. Taking into account the characteristics of the environment, an improved back propagation (BP) neural network is used to get the mapping relationships between the vibrations and the terrain types. Finally, classification testing has been conducted on five kinds of environments, including concrete, grassland, sand, gravel, and mixed. After 20 times random testing experiments, the classification accuracy has proven to be in the range 88.99%-100%, which verified the validity and the robustness of the algorithm and laid a foundation for the subsequent identification of terrain characteristic.

The future challenge for field robots is to increase the level of autonomy towards long distance (>1 km) and duration (>1h) applications. One of the key technologies is the ability to accurately estimate the properties of the traversed terrain to optimize onboard control strategies and energy efficient path-planning, ensuring safety and avoiding possible immobilization conditions that would lead to mission failure. Two main hypotheses are put forward in this research. The first hypothesis is that terrain can be effectively detected by relying exclusively on the measurement of quantities that pertain to the robot-ground interaction, i.e., on proprioceptive signals. Therefore, no visual or depth information is required. Then, artificial deep neural networks can provide an accurate and robust solution to the classification problem of different terrain types. Under these hypotheses, sensory signals are classified as time series directly by a Recurrent Neural Network or by a Convolutional Neural Network in the form of higher-level features or spectrograms resulting from additional processing. In both cases, results obtained from real experiments show comparable or better performance when contrasted with standard Support Vector Machine with the additional advantage of not requiring an a priori definition of the feature space.

Compact polarimetric SAR can preserve most of the information of full polarimetric SAR. The investigation of the information processing of polarimetric SAR interferometry is the hot spots in recent years. In this chapter, tree height retrieval and terrain classification application of compact PolInSAR are discussed more deeply.

Aiming at the problem of small sample size (SSS) and multimodal structure of hyperspectral remote sensing data, a feature extraction method of hyperspectral remote sensing images (HRSIs) combining direct linear discriminant analysis (LDA) and local Fisher discriminant analysis (LFDA) was proposed. Firstly, the local within-class scatter matrix and local between-class scatter matrix of the samples are calculated by LFDA method; then the projection matrix is solved by referring to the idea of direct LDA method, i.e., the local between-class scatter matrix is whitened first, and then the local within-class scatter matrix is diagonalized so that the zero-space of the local within-class scatter matrix which containing classification information is retained, so as to better maintaining the separability of the data while preserving the multimodal structure of data in the projection subspace. Finally, the minimum distance classifier is used to classify and recognize in the direct LFDA feature subspace. The experimental results of two real HRSIs show that compared with LDA, direct LDA and LFDA, the proposed direct LFDA algorithm can effectively improve the terrain recognition accuracy.

Robots are deployed to perform various tasks and assist mankind. This paper describes the operation of a four-legged robot, i.e., a quadruped in real-world scenarios, with multiple terrains and obstacles. This work focuses on the design and locomotion of a sprawling-type autonomous quadruped on flat surfaces, slopes, and stairs while simultaneously implementing obstacle avoidance.

The accurate terrain classification in real time is of great importance to an autonomous robot working in field, because the robot could avoid non-geometric hazards, adjust control scheme, or improve localization accuracy, with the aid of terrain classification. In this paper, we investigate the vibration-based terrain classification (VTC) in a dynamic environment, and propose a novel learning framework, named DyVTC, which tackles online-collected unlabeled data with concept drift. In the DyVTC framework, the exterior disagreement (ex-disagreement) and interior disagreement (in-disagreement) are proposed novely based on the feature diversity and intrinsic temporal correlation, respectively. Such a disagreement mechanism is utilized to design a pseudo-labeling algorithm, which shows its compelling advantages in extracting key samples and labeling, and consequently, the classification accuracy could be retrieved by incremental learning in a changing environment. Since two sets of features are extracted from frequency and time domain to generate disagreements, we also name the proposed method feature-temporal disagreement adaptation (FTDA). The real-world experiment shows that the proposed DyVTC could reach an accuracy of 89.5%, but the traditional time- and frequency-domain terrain classification methods could only reach 48.8% and 71.5%, respectively, in a dynamic environment.

Terrain knowledge around unmanned ground vehicles (UGVs) is vital for autonomous navigation. Having global understanding of the surroundings of UGVs is important, although the field of view from UGVs is very limited. Thus, we utilize an aerial vehicle to provide a large terrain map from sequential aerial images. In this paper, we present multiple techniques to accelerate the process of terrain classification so that it can run onboard on the aerial platform. The main techniques used to accelerate the process is a "knowledge distillation" of a deep neural net to a shallower one, and a super pixel implementation. We evaluated our system on Jetson TX1 with actual images collected from a weather balloon which confirmed the effectiveness of the proposed system.

IntroductionIn almost every planetary surface investigation, the characterization from a camera is a common initial step [1]. Mission Control is developing a science autonomy system called Autonomous Soil Assessment System: Contextualizing Rocks, Anomalies and Terrains in Exploratory Robotic Science (ASAS-CRATERS). It can enable automated surface characterization on planetary missions, which can benefit a wide range of science investigations and rover navigation alike. It can perform terrain classification and novelty detection using convolutional neural networks, and data aggregation to produce relevant data products for supporting science operations. Built on cutting-edge algorithms and off-the-shelf computing components, it offers low-cost ways to speed up tactical decision-making in next-generation commercial lunar missions.Background and MotivationAutonomy in Science OperationsSeveral factors are increasingly driving the need for autonomy in science operations. In traditional Mars rover operations, visual surface characterization and subsequent analysis and decision-making takes place in day-long tactical cycles [2]. Upcoming commercial lunar rover missions will have reduced latency, short lifetimes, and constrained bandwidth shared across several payloads. This will result in a need for rapid tactical decision-making processes with limited data, leaving little time for analysis, target identification, and making decisions. Payload operators may not receive data in a timely fashion, or worse, may not receive some data at all. Autonomous onboard terrain classification offers a way to downlink light-weight data products and reduce the bottleneck in scientific terrain assessment. Autonomous classification and novelty detection increase the chances of detecting novel/sparse features (e.g., lunar outcrop or pyroclasts) that may otherwise be missed or not downlinked when driving and other mission needs are prioritized.Application to Lunar GeologyWhile dedicated science instruments that reveal mineralogical and elemental composition improve our understanding of geological processes, a rover’s navigation sensors can document the morphology, morphometry, and composition of surface materials, regardless of primary investigation goals. High-resolution colour images and 3D data from stereo cameras provide information such as the size-frequency distribution and physical characteristics of craters and rocks, and regolith properties. All this offers valuable insight into the geologic setting. To provide a practical output as a science support tool for several types of missions, a classification scheme is being developed that segments a surface image into geological features that are visually distinct based on morphology, tone, and texture. This will be adapted for specific missions. See Figure 1 for a hand-labelled example.Figure 1: Hand-labelled lunar terrain classification example. Letters indicate crater degradation; P: Pristine; S: Semi-Degraded; G: Ghost. Right: original Yutu-1 image. Credit: CNSA.TechnologyAlgorithms and SoftwareASAS-CRATERS comprises three algorithms. First, the terrain classifier consists of a deep-learning encoder-decoder style network which classifies each image pixel into semantic terrain labels. Second, the novelty detector uses a semi-supervised convolutional neural network architecture with an autoencoder module and a binary classifier that work in series. Third, a data aggregator will combine the outputs on map tiles that are useable by onboard algorithms, lightweight for more efficient downlink, and enables faster backroom analysis and integration into GIS tools. See Figure 2 for a conceptual illustration of how ASAS-CRATERS outputs would be aggregated onto map tiles.Figure 2: Hand-made illustration of aggregating terrain classifier data on orthorectified map tiles.In the near future, additional algorithms will be developed to extend the capabilities of this autonomy system. This primarily includes an autonomous instrument targeting capability so that lunar science instruments, on rovers in particular, can use ASAS-CRATERS to identify and select features for targeting.Sensors and Computing HardwareASAS-CRATERS will be embedded on the Q8S, a high-performance, low-power Xilinx Zynq UltraScale+ System-on-Chip FPGA designed by Xiphos Technologies, which will fly in 2020. The Q8S consumes 3W at minimum and measures 90g and 80x80x22.3mm. Embedded on a flight-ready COTS system, ASAS-CRATERS can be rapidly integrated onto payload suites, rovers, and landers, offering low-cost advanced computing capabilities.Concept of OperationsPrior to deployment on a mission, ASAS-CRATERS’ machine learning algorithms will require training using relevant expert-labelled images from lunar and analogue datasets. Once a mission begins, the algorithms will be updated with images collected in situ. ASAS-CRATERS can then be used to classify terrain and detect features during nominal rover operations. The science team can integrate ASAS-CRATERS data products into their terrain analysis and make decisions on whether specific features merit deeper investigation using onboard instruments.Use Cases and BenefitsASAS-CRATERS can benefit science missions in several ways. First, it can support science operations in tactical cycles. Novelty detection can aid scientists that may miss features or spend valuable time in looking for them. The terrain classifier data products are low-dimensional representations which optimizes downlink. The classification itself can speed up scientific analysis in rapid tactical cycles and this becomes increasingly more useful in complex scenes diverse in mineralogy and lithology. Second, for high-priority features, onboard algorithms can perform instrument targeting and data triage for downlink prioritization. Third, as a semantically useful terrain representation, it can be used by advanced robotics algorithms to enable autonomous and intelligent navigation. Fourth, in human exploration architectures, ASAS-CRATERS can be embedded with crewed systems to provide autonomous capabilities that can support astronauts in geological field excursions.Field Tests and DemonstrationsThe terrain classifier was first developed by Mission Control under the Autonomous Soil Assessment System project [3]. In 2019, it was used to classify eight Mars-relevant terrain types in real-time at ~15 FPS (see Figure 3 for an example). This was a part of tests in Iceland under SAND-E (Semi-Autonomous Navigation for Detrital Environments), a NASA PSTAR funded project led by Dr. Ryan Ewing at Texas A&M University.Figure 3: Classifier output overlaid on one camera image during a SAND-E traverse in Iceland field tests.While ASAS-CRATERS is a multi-mission payload, near-term demonstrations are targeted for upcoming lunar missions in 2022 and 2023.AcknowledgementsThe authors would like to thank the Canadian Space Agency (CSA) for funding the development of ASAS-CRATERS and previous technology developments.References[1] Francis R. et al. (2014) SpaceOps. DOI: 10.2514/6.2014-1798. [2] Gaines D. et al. (2016) PlanRob, 115-125. [3] Faragalli M. et al. (2018) i-SAIRAS.

Aiming at the problems of existing terrain classification methods, this paper proposes a new terrain classification algorithm based on convolutional neural networks. First, the SegNet and SLIC superpixel segmentation methods are combined to solve the boundary blur problem in the terrain classification of the SegNet convolutional neural network. Then, through comparison experiments of irregular and complex terrain classification based on support vector machine, SegNet and SLIC-SegNet, the superiority of the new SLIC-SegNet terrain classification method is verified.

This research is aimed at investigating the interaction for a bipedal humanoid robot during locomotion on deformable terrains. The knowledge of deformable terrain properties has major implications in modelling the robot walking dynamics, which is the key to achieve stable gait patterns. We propose an in-situ terrain classification and estimation method in this paper to provide robots with such terrain properties. In particular, we introduce our method for classifying known terrains using recurrent neural networks (RNNs) and estimating the unknown terrain’s stiffness adopting the Bernstein-Goriatchkin model. We present our experimental procedures and analysis of the resulting data using NASA’s humanoid robot, Valkyrie, while it is performing a set of prespecified motions on different terrains, including gym foam mat, wood mulch, rubber mulch, and sand. It is concluded that using RNNs, we can efficiently classify the terrain types. For unknown terrains, our approach can estimate the terrain’s stiffness that demonstrates support forces under deformation.

The achievement of robot autonomy has environmental perception as a prerequisite. The hazards rendered from uneven, soft and slippery terrains, which are generally named non-geometric hazards, are another potential threat reducing the traversing efficient, and therefore receiving more and more attention from the robotics community. In the paper, the vibration-based terrain classification (VTC) is investigated by taking a very practical issue, i.e., lack of labels, into consideration. According to the intrinsic temporal correlation existing in the sampled terrain sequence, a modified Laplacian SVM is proposed to utilise the unlabelled data to improve the classification performance. To the best of our knowledge, this is the first paper studying semi-supervised learning problem in robotic terrain classification. The experiment demonstrates that: (1) supervised learning (SVM) achieves a relatively low classification accuracy if given insufficient labels, (2) feature-space homogeneity based semi-supervised learning (traditional Laplacian SVM) cannot improve supervised learning&rsquo, s accuracy, and even makes it worse, (3) feature- and temporal-space based semi-supervised learning (modified Laplacian SVM), which is proposed in the paper, could increase the classification accuracy very significantly.

Terrain classification is a hot topic in polarimetric synthetic aperture radar (PolSAR) image interpretation that aims at assigning a label to every pixel and forms a label matrix for a PolSAR image. From the perspective of human interpretation, classification is not in terms of pixels, but decomposed perceptual groups and structures. Therefore, a new perspective of label matrix completion is proposed to treat the classification task as a label matrix inversion process. Firstly, a matrix completion framework is built to avoid processing the large-scale PolSAR image data for traditional feature and classifier learning. Secondly, a light network is used to obtain the known labels for the complete task by uniform down-sampling of the entire image that aims to keep the shape of regions in a PolSAR image and reduce the computational complexity. Finally, the zeroth- and first-order label information is proposed as the prior distribution for label matrix completion to keep the structure and texture. The experiments are tested on real PolSAR images, which demonstrates that the proposed method can realize excellent classification results with less computation time and labeled pixels. Classification results with different down-sampling rates in the light network also prove the robustness of this method.

Recently the graph-based semi-supervised classification arouses vast amount of interest in remote sensing as it can utilize only a few labeled samples and large numbers of unlabeled samples to enhance the classification accuracy on various types of terrains. However in most of the conventional methods, multiple features (e.g. the scattering components, texture, color, etc) are concatenate together into a long vector for graph construction and classification. This not only ignores the physical attribute of features, but also causes so-called curse of dimensionality and limits the performances of classification. Inspired by the multi-view machine learning, we propose a spatial multi-attribute graph model and rank the attributes of polarimetric synthetic aperture radar (PolSAR) data for terrain classification in this paper. It firstly constructs multiple graphs according to the physical attributes of the groups of features based on different similarity metrics, then automatically optimizes a balanced weight for each graph and combines the spatial information between pixels for label propagation and classification. Since it takes into account of the physical properties from PolSAR data for feature fusion and graph construction, the mechanically feature stacking and curse of dimensionality can be avoided. Experimental results on synthesized PolSAR data and real ones show enhanced classification accuracy of the proposed method compared with state-of-the-art graph-based methods when only a small number of labeled samples are available. Our empirical studies also indicate that the covariance matrix play predominant roles for PolSAR classification.

It is well known that various features extraction approaches are utilized in polarimetric synthetic aperture (PolSAR) terrain classification for representing the data characteristic. It needs relev...

Micro-terrain and micro-weather have an important impact on transmission line galloping. In order to carry out galloping prediction of micro-terrain, the classification of galloping micro-terrain is studied in this work. Firstly, we collect historical data of 1537 galloping points from the State Grid Corporation of China, and select 208 galloping points located in the micro-terrain area by analyzing the altitude and the topographic relief characteristics around each galloping point. Then the galloping micro-terrain types are extracted by Empirical Orthogonal Function method, the first four spatial modes of galloping micro-terrain are the windward slope of east-west mountain area, the windward slope of north-south mountain area, the independent hill, and the saddle back of mountain/hill. Finally, the regional characteristics of typical micro-terrain are analyzed according to the actual lines.

We propose a lightweight and fast learning algorithm for classifying the features of an unknown terrain that a robot is navigating in. Most of the existing research on unknown terrain classification by mobile robots relies on a single powerful classifier to correctly identify the terrain using sensor data from a single sensor like laser or camera. In contrast, our proposed approach uses multiple modalities of sensed data and multiple, weak but less-complex classifiers for classifying the terrain types. The classifiers are combined using an ensemble learning algorithm to improve the algorithm’s training rate as compared to an individual classifier. Our algorithm was tested with data collected by navigating a four-wheeled, autonomous robot, called Explorer, over different terrains including brick, grass, rock, sand, and concrete. Our results show that our proposed approach performs better with up to 63% better prediction accuracy for some terrains as compared to a support vector machine (SVM)-based learning technique that uses sensor data from a single sensor. Despite using multiple classifiers, our algorithm takes only a fraction (1/65) of the time on average, as compared to the SVM technique.

Terrain classification is a critical component of any autonomous mobile robot system operating in unknown real-world environments. Over the years, several proprioceptive terrain classification techniques have been introduced to increase robustness or act as a fallback for traditional vision based approaches. However, they lack widespread adaptation due to various factors that include inadequate accuracy, robustness and slow run-times. In this paper, we use vehicle-terrain interaction sounds as a proprioceptive modality and propose a deep long-short term memory based recurrent model that captures both the spatial and temporal dynamics of such a problem, thereby overcoming these past limitations. Our model consists of a new convolution neural network architecture that learns deep spatial features, complemented with long-short term memory units that learn complex temporal dynamics. Experiments on two extensive datasets collected with different microphones on various indoor and outdoor terrains demonstrate state-of-the-art performance compared to existing techniques. We additionally evaluate the performance in adverse acoustic conditions with high-ambient noise and propose a noise-aware training scheme that enables learning of more generalizable models that are essential for robust real-world deployments.

The ability to precisely classify different types of terrain is extremely important for Unmanned Aerial Vehicles (UAVs). There are multiple situations in which terrain classification is fundamental for achieving a UAV’s mission success, such as emergency landing, aerial mapping, decision making, and cooperation between UAVs in autonomous navigation. Previous research works describe different terrain classification approaches mainly using static features from RGB images taken onboard UAVs. In these works, the terrain is classified from each image taken as a whole, not divided into blocks; this approach has an obvious drawback when applied to images with multiple terrain types. This paper proposes a robust computer vision system to classify terrain types using three main algorithms, which extract features from UAV’s downwash effect: Static textures- Gray-Level Co-Occurrence Matrix (GLCM), Gray-Level Run Length Matrix (GLRLM) and Dynamic textures- Optical Flow method. This system has been fully implemented using the OpenCV library, and the GLCM algorithm has also been partially specified in a Hardware Description Language (VHDL) and implemented in a Field Programmable Gate Array (FPGA)-based platform. In addition to these feature extraction algorithms, a neural network was designed with the aim of classifying the terrain into one of four classes. Lastly, in order to store and access all the classified terrain information, a dynamic map, with this information was generated. The system was validated using videos acquired onboard a UAV with an RGB camera.

While controlling a lower limb exoskeleton providing walking assistance to wearers, the walking terrain is an important factor that should be considered for meeting performance and safety requirements. Therefore, we developed a method to estimate the slope and elevation using the contact points between the limb exoskeleton and ground. We used the center of pressure as a contact point on the ground and calculated the location of the contact points on the walking terrain based on kinematic analysis of the exoskeleton. Then, a set of contact points collected from each step during walking was modeled as the plane that represents the surface of the walking terrain through the least-square method. Finally, by comparing the normal vectors of the modeled planes for each step, features of the walking terrain were estimated. We analyzed the estimation accuracy of the proposed method through experiments on level ground, stairs, and a ramp. Classification using the estimated features showed recognition accuracy higher than 95% for all experimental motions. The proposed method approximately analyzed the movement of the exoskeleton on various terrains even though no prior information on the walking terrain was provided. The method can enable exoskeleton systems to actively assist walking in various environments.

The ability to autonomously navigate in an unknown and dynamic environment avoiding obstacles while, at the same time, classify various types of terrain are challenges that have been mainly faced by researchers from the computer vision area. Solutions to these problems are of great interest for collaborative autonomous navigation robots. For example, an Unmanned Aerial Vehicle (UAV) may be used to determine the path which an Unmanned Surface Vehicle (USV) has to navigate to reach the intended destination. This paper presents a novel vision-based algorithm that allows for independent navigation with on flight obstacle avoidance planning, while simultaneously classifying different terrain types using a Wiener-Khinchin (W-K) Filter.

The objective of this study was to investigate if the inertial data collected from normal human walk can be used to reveal the underlying terrain types. For this purpose, we recorded the gait patterns of normal human walk on six different terrain types with variation in hardness and friction using body mounted inertial sensors. We collected accelerations and angular velocities of 40 healthy subjects with two smartphones embedded inertial measurement units (MPU-6500) attached at two different body locations (chest and lower back). The recorded data were segmented with stride based segmentation approach and 194 tempo-spectral features were computed for each stride. We trained two machine learning classifiers, namely random forest and support vector machine, and cross validated the results with 10-fold cross-validation strategy. The classification tasks were performed on indoor&ndash, outdoor terrains, hard&ndash, soft terrains, and a combination of binary, ternary, quaternary, quinary and senary terrains. From the experimental results, the classification accuracies of 97% and 92% were achieved for indoor&ndash, outdoor and hard&ndash, soft terrains, respectively. The classification results for binary, ternary, quaternary, quinary and senary class classification were 96%, 94%, 92%, 90%, and 89%, respectively. These results demonstrate that the stride data collected with the low-level signals of a single IMU can be used to train classifiers and predict terrain types with high accuracy. Moreover, the problem at hand can be solved invariant of sensor type and sensor location.

In this work, we propose to use learned features for terrain classification of Polarimetric Synthetic Aperture Radar (PolSAR) images. In the proposed classification framework, the learned features are extracted from sliding window regions using Convolutional Neural Networks (CNNs), and then they are used for the classification with the linear Support Vector Machine (SVM) classifier. The classification performance of the proposed approach is compared with numerous target decomposition theorems (TDs) as the engineered features tested with two classifiers: Collective Network of Binary Classifiers (CNBCs) and SVMs. The experimental evaluations over two commonly used benchmark AIRSAR PolSAR images, San Francisco Bay and Flevoland at L-Band, reveal that the classification performance of the learned features with CNNs outperforms the performance of the engineered features as TDs even the dimension of learned features is the quarter of the engineered features. acceptedVersion

This work was partially funded by FCT Strategic Program UID/EEA/00066/203 of the Center of Technologies and System (CTS) of UNINOVA - Institute for the Development of new Technologies. Knowing how to identify terrain types is especially important in the autonomous navigation, mapping, decision making and detect landings areas. A recent area is in cooperation and improvement of autonomous behavior between robots. For example, an unmanned aerial vehicle (UAV) is used to identify a possible landing area or used in cooperation with other robots to navigate in unknown terrains. This paper presents a computer vision algorithm capable of identifying the terrain type where the UAV is flying, using its rotors’ downwash effect. The algorithm is a fusion between the frequency Wiener-Khinchin adapted and spatial Empirical Mode Decomposition (EMD) domains. In order to increase certainty in terrain identification, machine learning is also used. The system is validated using videos acquired onboard of a UAV with an RGB camera. authorsversion published

In this study, we propose adaptive locomotion for an autonomous multilegged walking robot, an image infilling method for terrain classification based on a combination of speeded up robust features, and binary robust invariant scalable keypoints (SURF-BRISK). The terrain classifier is based on the bag-of-words (BoW) model and SURF-BRISK, both of which are fast and accurate. The image infilling method is used for identifying terrain with obstacles and mixed terrain, their features are magnified to help with recognition of different complex terrains. Local image infilling is used to improve low accuracy caused by obstacles and super-pixel image infilling is employed for mixed terrain. A series of experiments including classification of terrain with obstacles and mixed terrain were conducted and the obtained results show that the proposed method can accurately identify all terrain types and achieve adaptive locomotion.

In this paper we propose two novel deep learning-based terrain classification methods robust to illumination changes. The use of cameras is challenged by a variety of factors, of most importance being the changes in illumination. On the other hand, since the temperature of various types of terrains depends on the thermal characteristics of the terrain, the terrain classification can be aided by utilizing the thermal information in addition to visible information. Thus we propose 'TU-Net (Two U-Net)' based on the U-Net and 'TDeepLab (Two DeepLab)' based on DeepLab, which combine visible and thermal images and train the network robust to illumination changes implicitly. To improve the network's learning capability, we expand the proposed methods to the Siamese-based method, which explicitly trains the network to be robust to illumination changes. We also investigate multiple options to fuse the visible and thermal images at at the bottom layer, middle layer, or the top layer of the network. We evaluate the proposed methods with a challenging new dataset consisting of visible and thermal images, which were collected from 10 am till 5 pm (after sunset), and we show the effectiveness of the proposed methods.

UID/EEA/00066/203 knowing how to identify terrain types is especially important in the autonomous navigation, mapping, decision making and emergency landings areas. For example, an unmanned aerial vehicle (UAV) can use it to find a suitable landing position or to cooperate with other robots to navigate across an unknown region. Previous works on terrain classification from RGB images taken onboard of UAVs shown that only static pixel-based features were tested with a considerable classification error. This paper presents a computer vision algorithm capable of identifying the terrain from RGB images with improved accuracy. The algorithm complement the static image features and dynamic texture patterns produced by UAVs rotors downwash effect (visible at lower altitudes) and machine learning methods to classify the underlying terrain. The system is validated using videos acquired onboard of a UAV with a RGB camera. publishersversion published

An in-depth understanding of the nature of the available terrain and its exposure to avalanche hazard is crucial for making informed risk management decisions when travelling in the backcountry. While the Avalanche Terrain Exposure Scale (ATES) is broadly used for providing recreationists with terrain information, this type of terrain classification has so far only seen limited adoption within the professional ski guiding community. We hypothesize that it is the generic nature and small number of terrain classes of ATES and its precursor systems that prevent them from offering meaningful assistance to professional decision makers. Working with two mechanized skiing operations in British Columbia, Canada, we present a new approach for deriving terrain classifications from daily terrain assessment records. We used a combination of self-organizing maps and hierarchical clustering to identify groups of ski runs that have been assessed similarly in the past and organized them into operation-specific ski run hierarchies. We then examined the nature of the emerging ski run hierarchies using comprehensive run characterizations from experienced guides. Our approach produces high-resolution ski run hierarchies that offer a more nuanced and meaningful perspective on the available skiing terrain and provide new opportunities for examining professional avalanche risk management practices and developing meaningful decision aids.

With the improving imaging technology of the SAR, more and more high-resolution SAR images are obtained. Interpreting SAR images manually is a vast task and may lead to many mistakes. Therefore, it is greatly necessary to develop the corresponding automatic algorithms. Focusing on the target detection, many algorithms have been developed.

The heterogenous areas of the polarimetric synthetic aperture radar (PolSAR) image are hardly to be classified into semantic homogenous regions due to the complex terrain structures. In order to overcome these disadvantages, a PolSAR image classification method is proposed based on the multi-feature learning and the topic model. The proposed method makes use of three kinds of features to formulate the visual codewords. Then, the higher level features are learned by the topic model for classification. Experimental results illustrate that the proposed method can obtain better performance than the state-of-art methods especially for the heterogenous areas.

In the previous chapters, road terrain classification using an accelerometer, a camera and a downward facing LRF has been investigated.

The utilization of an accelerometer is supposed to estimate the road profile from which the spatial features are extracted. However, the estimated road profile only represents the curvature of the terrain on the vehicle’s moving orientation. Here, it is proposed to employ the LRF sensor to represent the road surface which would contain substantial information. Moreover, because it works with higher scanning speed, this classification method is expected to be minimally affected by the change of the vehicle’s speed.

There has been a growing interest in SAR image interpretation for a variety of civilian and military applications, such as terrain classification, land or sea monitoring, target detection and recognition, etc. Presently, one of major strategies of SAR image processing is to use the classical methods of statistical pattern recognition, where it is crucial to develop precise models for the statistics of the pixel amplitudes or intensities (Oliver and Quegan in Understanding Synthetic Aperture Radar Images. Artech House, Norwood 1998, [1]).

In this paper, improved scattering parameters of polarimetric synthetic aperture radar (POLSAR) image based on spatial information and Bayes rule is proposed. The spatial information of scattering parameters is introduced by using an adaptive weight window. Bayes rule is used to improve the performance of the scattering parameters. Experiments on real AIRSAR L-band fully POLSAR data are carried out, and the efficacy of the improved scattering parameters is verified.

The images of different road terrain types present abundant information and significantly different characteristics. Hence, it is considered to use a camera to capture images of road surfaces in an attempt to overcome the issues found using accelerometer data only. By processing and extracting features from those images, road terrain classification is expected to be improved.

This paper deals with the terrain classification problem for autonomous service robots in semi-structured outdoor environments. The aim is to recognize the drivable terrain in front of a robot that navigates on roads of different surfaces, avoiding areas that are considered non-drivable. Since the system should be robust to such factors as changing lighting conditions, mud and fallen leaves, we employ multi-sensor perception with a monocular camera and a 2D laser scanner. The labeling of the terrain obtained from a Random Trees classifier is refined by context-aware inference using the Conditional Random Field. We demonstrate that automatic learning of the parameters for Conditional Random Fields improves results in comparison to similar approaches without the context-aware inference or with parameters set by hand.