Search

Neuromorphic vision sensors such as the Dynamic and Active-pixel Vision Sensor (DAVIS) using silicon retina are inspired by biological vision, they generate streams of asynchronous events to indicate local log-intensity brightness changes. Their properties of high temporal resolution, low-bandwidth, lightweight computation, and low-latency make them a good fit for many applications of motion perception in the intelligent vehicle. However, as a younger and smaller research field compared to classical computer vision, neuromorphic vision is rarely connected with the intelligent vehicle. For this purpose, we present three novel datasets recorded with DAVIS sensors and depth sensor for the distracted driving research and focus on driver drowsiness detection, driver gaze-zone recognition, and driver hand-gesture recognition. To facilitate the comparison with classical computer vision, we record the RGB, depth and infrared data with a depth sensor simultaneously. The total volume of this dataset has 27360 samples. To unlock the potential of neuromorphic vision on the intelligent vehicle, we utilize three popular event-encoding methods to convert asynchronous event slices to event-frames and adapt state-of-the-art convolutional architectures to extensively evaluate their performances on this dataset. Together with qualitative and quantitative results, this work provides a new database and baseline evaluations named NeuroIV in cross-cutting areas of neuromorphic vision and intelligent vehicle.

Dexterous manipulation in robotic hands relies on an accurate sense of artificial touch. Here we investigate neuromorphic tactile sensation with an event-based optical tactile sensor combined with spiking neural networks for edge orientation detection. The sensor incorporates an event-based vision system (mini-eDVS) into a low-form factor artificial fingertip (the NeuroTac). The processing of tactile information is performed through a Spiking Neural Network with unsupervised Spike-Timing-Dependent Plasticity (STDP) learning, and the resultant output is classified with a 3-nearest neighbours classifier. Edge orientations were classified in 10-degree increments while tapping vertically downward and sliding horizontally across the edge. In both cases, we demonstrate that the sensor is able to reliably detect edge orientation, and could lead to accurate, bio-inspired, tactile processing in robotics and prosthetics applications.

Abnormal event detection is an important task in research and industrial applications, which has received considerable attention in recent years. Existing methods usually rely on standard frame-based cameras to record the data and process them with computer vision technologies. In contrast, this paper presents a novel neuromorphic vision based abnormal event detection system. Compared to the frame-based camera, neuromorphic vision sensors, such as Dynamic Vision Sensor (DVS), do not acquire full images at a fixed frame rate but rather have independent pixels that output intensity changes (called events ) asynchronously at the time they occur. Thus, it avoids the design of the encryption scheme. Since events are triggered by moving edges on the scene, DVS is a natural motion detector for the abnormal objects and automatically filters out any temporally-redundant information. Based on this unique output, we first propose a highly efficient method based on the event density to select activated event cuboids and locate the foreground. We design a novel event -based multiscale spatio-temporal descriptor to extract features from the activated event cuboids for the abnormal event detection. Additionally, we build the NeuroAED dataset, the first public dataset dedicated to abnormal event detection with neuromorphic vision sensor. The NeuroAED dataset consists of four sub-datasets: Walking, Campus, Square, and Stair dataset. Experiments are conducted based on these datasets and demonstrate the high efficiency and accuracy of our method.

Neurorobotic mimics the structural and functional principles of living creature systems. Modeling a single system by robotic hardware and software has existed for decades. However, an integrated toolset studying the interaction of all systems has not been demonstrated yet. We present a hybrid neuromorphic computing paradigm to bridge this gap by combining the neurorobotics platform (NRP) with the neuromorphic snake-like robot (NeuroSnake). This paradigm encompasses the virtual models, neuromorphic sensing and computing capabilities, and physical bio-inspired bodies, with which an experimenter can design and execute both in-silico and in-vivo robotic experimentation easily. The NRP is a public Web-based platform for easily testing brain models with virtual bodies and environments. The NeuroSnake is a bio-inspired robot equipped with a silico-retina sensor and neuromorphic computer for power-efficiency applications. We illustrate the efficiencies of our paradigm with an easy designing of a visual pursuit experiment in the NRP. We study two automatic behavior learning tasks which are further integrated into a complex task of semi-autonomous pole climbing. The result shows that robots could build new learning rules in a less explicit manner inspired by living creatures. Our method gives an alternative way to efficiently develop complex behavior control of the ro As spiking neural network is a bio-inspired neural network and the NeuroSnake robot is equipped with a spike-based silicon retina camera, the control system can be easily implemented via spiking neurons simulated on neuromorphic hardware, such as SpiNNaker.bot.

The cameras in modern gaze-tracking systems suffer from fundamental bandwidth and power limitations, constraining data acquisition speed to 300 Hz realistically. This obstructs the use of mobile eye trackers to perform, e.g., low latency predictive rendering, or to study quick and subtle eye motions like microsaccades using head-mounted devices in the wild. Here, we propose a hybrid frame-event-based near-eye gaze tracking system offering update rates beyond 10,000 Hz with an accuracy that matches that of high-end desktop-mounted commercial trackers when evaluated in the same conditions. Our system, previewed in Figure 1, builds on emerging event cameras that simultaneously acquire regularly sampled frames and adaptively sampled events. We develop an online 2D pupil fitting method that updates a parametric model every one or few events. Moreover, we propose a polynomial regressor for estimating the point of gaze from the parametric pupil model in real time. Using the first event-based gaze dataset, we demonstrate that our system achieves accuracies of 0.45°-1.75° for fields of view from 45° to 98°. With this technology, we hope to enable a new generation of ultra-low-latency gaze-contingent rendering and display techniques for virtual and augmented reality.

Restoring tactile sensation is essential to enable in-hand manipulation and the smooth, natural control of upper-limb prosthetic devices. Here we present a platform to contribute to that long-term vision, combining an anthropomorphic robot hand (QB SoftHand) with a neuromorphic optical tactile sensor (neuroTac). Neuromorphic sensors aim to produce efficient, spike-based representations of information for bio-inspired processing. The development of this 5-fingered, sensorized hardware platform is validated with a customized mount allowing manual control of the hand. The platform is demonstrated to succesfully identify 4 objects from the YCB object set, and accurately discriminate between 4 directions of shear during stable grasps. This platform could lead to wide-ranging developments in the areas of haptics, prosthetics and telerobotics.

In this paper, we proposed a novel zero velocity detector, the Dynamic-Vision-Sensor (DVS)-aided Stance Phase Optimal dEtection (SHOE) detector, for Zero-velocity-UPdaTe (ZUPT)-aided Inertial Navigation Systems (INS) augmented by a foot-mounted event-based camera DVS128. We observed that the firing rate of the DVS consistently increased during the swing phase and decreased during the stance phase in indoor walking experiments. We experimentally determined that the optimal placement configuration for zero-velocity detection is to mount the DVS next to an Inertial Measurement Unit (IMU) and face the sensor outward. The DVS-SHOE detector was derived in a General Likelihood Ratio Test (GLRT) framework, combining statistics of the conventional SHOE detector and the DVS firing rate. This paper used two methods to evaluate the proposed DVS-SHOE detector. First, we compared the detection performances of the SHOE detector and the DVS-SHOE detector. The experimental results showed that the DVS-SHOE detector achieved a lower false alarm rate than the SHOE detector. Second, we compared the navigation performance of the ZUPT-aided INS using the SHOE detector and the DVS detector. The experimental results showed that the Circular Error Probable (CEP) of the case using DVS-SHOE was reduced by around 25 % from 1.2 m to 0.9 m, as compared to the case of the SHOE detector.

Vector Symbolic Architectures belong to a family of related cognitive modeling approaches that encode symbols and structures in high-dimensional vectors. Similar to human subjects, whose capacity to process and store information or concepts in short-term memory is subject to numerical restrictions, the capacity of information that can be encoded in such vector representations is limited and one way of modeling the numerical restrictions to cognition. In this paper, we analyze these limits regarding information capacity of distributed representations. We focus our analysis on simple superposition and more complex, structured representations involving convolutive powers to encode spatial information. In two experiments, we find upper bounds for the number of concepts that can effectively be stored in a single vector only depending on the dimensionality of the underlying vector space.

We propose a neuromorphic approach to perception, reasoning and motor control using Spiking Neural Networks in mobile robotics. We demonstrate this by using a mobile robotic manipulator solving a pick-and-place task. All sensory data is provided by spike-based silicon retina cameras - eDVS (embedded Dynamic Vision Sensor) - and all reasoning and motor control is implemented in Spiking Neural Networks. For the given scenario, the robot is capable of detecting a sequence of objects blinking at different frequencies, finding one object that is not in the right place of the sequence, picking up this object and moving it to its correct position. Such a scenario demonstrates how to build large-scale networks solving a high-level cognitive task by combining several smaller networks responsible for low-level tasks. Importantly, here we focus only on generating a neural network that is capable of performing the task. This will be the basis of future work using neural network learning algorithms to improve task performance. The long-term goal is to learn sophisticated behaviours by experience while at the same time being able to introduce expert knowledge for intermediate tasks that can be used to initialize the network or to speed up the learning process.

Factor graphs are probabilistic graphical frameworks for modeling complex and dynamic systems. They can be used in a broad range of application domains, from machine learning and robotics, to signal processing and digital communications. One important aspect that makes a factor graph very useful and very promising to be applied widely is its inference mechanism that is suitable for performing a complex model-based reasoning. However, its features have not fully explored and factor graphs are still used mainly as modeling tools that run on standard computers. Whereas in real applications such as robotics, one needs a practical implementation of such a framework. In this paper, we describe the development of a factor-graph-based inference engine that runs on a System-on-Chip (SoC). Running natively on a low level hardware, our factor graph engine delivers highest performance for real-time applications. We designed the embedded architecture so that it conveys important aspects such as modularity, scalability, flexibility and platform-friendly framework. The proposed architecture has customizable levels of parallelism as well as re-configurable modules that are extensible to accommodate large networks. We optimized the design to achieve high efficiency in terms of clock latency and resources consumption. We have tested our design on Xilinx Zynq-7000 SoCs and the implementation result demonstrates that the proposed framework can potentially be extended into a massively distributed probabilistic computing engine.

Predicting future behavior of other traffic participants is an essential task that needs to be solved by automated vehicles and human drivers alike to achieve safe and situation-aware driving. Modern approaches to vehicles trajectory prediction typically rely on data-driven models like neural networks, in particular LSTMs (Long Short-Term Memorys), achieving promising results. However, the question of optimal composition of the underlying training data has received less attention. In this paper, we expand on previous work on vehicle trajectory prediction based on neural network models employing distributed representations to encode automotive scenes in a semantic vector substrate. We analyze the influence of variations in the training data on the performance of our prediction models. Thereby, we show that the models employing our semantic vector representation outperform the numerical model when trained on an adequate data set and thereby, that the composition of training data in vehicle trajectory prediction is crucial for successful training. We conduct our analysis on challenging real-world driving data.

With the advanced development of image processing technology, visible light positioning (VLP) system based on image sensors has attracted more and more attention. However, as a commonly used light receiver, traditional CMOS camera has limited dynamic range and high latency, which is susceptible to various lighting and environmental factors. Moreover, high computational cost from image processing is unavoidable for most of visible light positioning systems. In our work, a novel VLP system using an event-based neuromorphic vision sensor (event camera) as the light receiver is proposed. Due to the low latency and microsecond-level temporal resolution of the event camera, our VLP system is able to identify multiple high-frequency flickering LEDs in asynchronous events simultaneously leaving out the need for data association and traditional image processing methods. A multi-LED fusion method is applied and a high positioning accuracy of 3cm is achieved when the height between LEDs and the event camera is within 1m.

Drowsiness driving is a principal factor of many fatal traffic accidents. This paper presents the first event-based drowsiness driving detection (EDDD) system by using the recently developed neuromorphic vision sensor. Compared with traditional frame-based cameras, neuromorphic vision sensors, such as Dynamic Vision Sensors (DVS), have a high dynamic range and do not acquire full images at a fixed frame rate but rather have independent pixels that output intensity changes (called events) asynchronously at the time they occur. Since events are generated by moving edges in the scene, DVS is considered as an efficient and effective detector for the drowsiness driving-related motions. Based on this unique output, this work first proposes a highly efficient method to recognize and localize the driver’s eyes and mouth motions from event streams. We further design and extract event-based drowsiness-related features directly from the event streams caused by eyes and mouths motions, then the EDDD model is established based on these features. Additionally, we provide the EDDD dataset, the first public dataset dedicated to event-based drowsiness driving detection. The EDDD dataset has 260 recordings in daytime and evening with several challenging scenes such as subjects wearing glasses/sunglasses. Experiments are conducted based on this dataset and demonstrate the high efficiency and accuracy of our method under different illumination conditions. As the first investigation of the usage of DVS in drowsiness driving detection applications, we hope that this work will inspire more event-based drowsiness driving detection research.

As a bio-inspired and emerging sensor, an event-based neuromorphic vision sensor has a different working principle compared to the standard frame-based cameras, which leads to promising properties of low energy consumption, low latency, high dynamic range (HDR), and high temporal resolution. It poses a paradigm shift to sense and perceive the environment by capturing local pixel-level light intensity changes and producing asynchronous event streams. Advanced technologies for the visual sensing system of autonomous vehicles from standard computer vision to event-based neuromorphic vision have been developed. In this tutorial-like article, a comprehensive review of the emerging technology is given. First, the course of the development of the neuromorphic vision sensor that is derived from the understanding of biological retina is introduced. The signal processing techniques for event noise processing and event data representation are then discussed. Next, the signal processing algorithms and applications for event-based neuromorphic vision in autonomous driving and various assistance systems are reviewed. Finally, challenges and future research directions are pointed out. It is expected that this article will serve as a starting point for new researchers and engineers in the autonomous driving field and provide a bird's-eye view to both neuromorphic vision and autonomous driving research communities.

Predicting future behavior and positions of other traffic participants from observations is a key problem that needs to be solved by human drivers and automated vehicles alike to safely navigate their environment and to reach their desired goal. In this paper, we expand on previous work on an automotive environment model based on vector symbolic architectures (VSAs). We investigate a vector-representation to encapsulate spatial information of multiple objects based on a convolutive power encoding. Assuming that future positions of vehicles are influenced not only by their own past positions and dynamics (e.g., velocity and acceleration) but also by the behavior of the other traffic participants in the vehicle's surroundings, our motivation is 3-fold: we hypothesize that our structured vector-representation will be able to capture these relations and mutual influence between multiple traffic participants. Furthermore, the dimension of the encoding vectors remains fixed while being independent of the number of other vehicles encoded in addition to the target vehicle. Finally, a VSA-based encoding allows us to combine symbol-like processing with the advantages of neural network learning. In this work, we use our vector representation as input for a long short-term memory (LSTM) network for sequence to sequence prediction of vehicle positions. In an extensive evaluation, we compare this approach to other LSTM-based benchmark systems using alternative data encoding schemes, simple feed-forward neural networks as well as a simple linear prediction model for reference. We analyze advantages and drawbacks of the presented methods and identify specific driving situations where our approach performs best. We use characteristics specifying such situations as a foundation for an online-learning mixture-of-experts prototype, which chooses at run time between several available predictors depending on the current driving situation to achieve the best possible forecast.

In this paper we describe the NeuroRobotic Mouse (NeRmo) a low-cost, modular bio mimetic robot, mimicking the actuation and walking behaviour of a common mouse (Mus musculus). This latest version has 13 Degrees of Freedom with 21 tendon driven joints and can be controlled in both open and closed loop. It is capable of different gaits as well as keeping the body upright when in a sitting position. The robot includes joint position sensors, pressure sensors on the soles of the feet as well as two cameras in the head. As the design of this robotic platform was inspired by detailed observations of the biomechanics of mice and rats, it can be used in motion research using animal data as am element of comparison, or even as actuation input.

Anthropomimetic robots sense, behave, interact, and feel like humans. By this definition, they require human-like physical hardware and actuation but also brain-like control and sensing. The most self-evident realization to meet those requirements would be a human-like musculoskeletal robot with a brain-like neural controller. While both musculoskeletal robotic hardware and neural control software have existed for decades, a scalable approach that could be used to build and control an anthropomimetic human-scale robot has not yet been demonstrated. Combining Myorobotics, a framework for musculoskeletal robot development, with SpiNNaker, a neuromorphic computing platform, we present the proof of principle of a system that can scale to dozens of neurally controlled, physically compliant joints. At its core, it implements a closed-loop cerebellar model that provides real-time, low-level, neural control at minimal power consumption and maximal extensibility. Higher-order (e.g., cortical) neural networks and neuromorphic sensors like silicon retinae or cochleae can be incorporated.

Predicting future motion of other vehicles or, more generally, the development of traffic situations, is an essential step towards secure, context-aware automated driving. On the one hand, human drivers are able to anticipate driving situations continuously based on the currently perceived behavior of other traffic participants while incorporating prior experience. On the other hand, the most successful data-driven prediction models are typically trained on large amounts of recorded data before deployment achieving remarkable results. In this paper, we present a mixture-of-experts online learning model encapsulating both ideas. Our system learns at run time to choose between several models, which have been previously trained offline, based on the current situational context. We show that our model is able to improve over the offline models already after a short ramp-up phase. We evaluate our system on real world driving data.

Objective. The objective of this work is to use the capability of spiking neural networks to capture the spatio-temporal information encoded in time-series signals and decode them without the use of hand-crafted features and vector-based learning and the realization of the spiking model on low-power neuromorphic hardware. Approach. The NeuCube spiking model was used to classify different grasp movements directly from raw surface electromyography signals (sEMG), the estimations of the applied finger forces as well as the classification of two motor imagery movements from raw electroencephalography (EEG). In a parallel investigation, the designed spiking decoder was implemented on SpiNNaker neuromorphic hardware, which allows low-energy real-time processing. Main results. Experimental results reveal a better classification accuracy using the NeuCube model compared to traditionalmachine learning methods. For sEMG classification, we reached a training accuracy of 85% and a test accuracy of 84.8%, as well as less than 19% of relative root mean square error (rRMSE) when estimating finger forces from six subjects. For the EEG classification, a mean accuracy of 75% was obtained when tested on raw EEG data from nine subjects from the existing 2b dataset from ‘BCI competition IV’. Significance. This work provides a proof of concept for a successful implementation of the NeuCube spiking model on the SpiNNakerneuromorphic platform for raw sEMG and EEG decoding, which could chart a route ahead for a new generation of portable closed-loop and low-power neuroprostheses.

A neuromorphic vision sensors is a novel passive sensing modality and frameless sensors with several advantages over conventional cameras. Frame-based cameras have an average frame-rate of 30 fps, causing motion blur when capturing fast motion, e.g., hand gesture. Rather than wastefully sending entire images at a fixed frame rate, neuromorphic vision sensors only transmit the local pixel-level changes induced by the movement in a scene when they occur. This leads to advantageous characteristics, including low energy consumption, high dynamic range, a sparse event stream and low response latency. In this study, a novel representation learning method was proposed: Fixed Length Gists Representation (FLGR) learning for event-based gesture recognition. Previous methods accumulate events into video frames in a time duration (e.g., 30 ms) to make the accumulated image-level representation. However, the accumulated-frame-based representation waives the friendly event-driven paradigm of neuromorphic vision sensor. New representation are urgently needed to fill the gap in non-accumulated-frame-based representation and exploit the further capabilities of neuromorphic vision. The proposed FLGR is a sequence learned from mixture density autoencoder and preserves the nature of event-based data better. FLGR has a data format of fixed length, and it is easy to feed to sequence classifier. Moreover, an RNN-HMM hybrid was proposed to address the continuous gesture recognition problem. Recurrent neural network (RNN) was applied for FLGR sequence classification while hidden Markov model (HMM) is employed for localizing the candidate gesture and improving the result in a continuous sequence. A neuromorphic continuous hand gestures dataset (Neuro ConGD Dataset) was developed with 17 hand gestures classes for the community of the neuromorphic research. Hopefully, FLGR can inspire the study on the event-based highly efficient, high-speed, and high-dynamic-range sequence classification tasks.

Non-invasive, electroencephalography (EEG)-based brain-computer interfaces (BCIs) on motor imagery movements translate the subject’s motor intention into control signals through classifying the EEG patterns caused by different imagination tasks, e.g., hand movements. This type of BCI has been widely studied and used as an alternative mode of communication and environmental control for disabled patients, such as those suffering from a brainstem stroke or a spinal cord injury (SCI). Notwithstanding the success of traditional machine learning methods in classifying EEG signals, these methods still rely on hand-crafted features. The extraction of such features is a difficult task due to the high non-stationarity of EEG signals, which is a major cause by the stagnating progress in classification performance. Remarkable advances in deep learning methods allow end-to-end learning without any feature engineering, which could benefit BCI motor imagery applications. We developed three deep learning models: (1) A long short-term memory (LSTM), (2) a spectrogram-based convolutional neural network model (CNN), and (3) a recurrent convolutional neural network (RCNN), for decoding motor imagery movements directly from raw EEG signals without (any manual) feature engineering. Results were evaluated on our own publicly available, EEG data collected from 20 subjects and on an existing dataset known as 2b EEG dataset from “BCI Competition IV”. Overall, better classification performance was achieved with deep learning models compared to state-of-the art machine learning techniques, which could chart a route ahead for developing new robust techniques for EEG signal decoding. We underpin this point by demonstrating the successful real-time control of a robotic arm using our CNN based BCI.

Objective The objective of this work is to present gumpy, a new free and open source Python toolbox designed for hybrid brain-computer interface (BCI). Approach Gumpy provides state-of-the-art algorithms and includes a rich selection of signal processing methods that have been employed by the BCI community over the last 20 years. In addition, a wide range of classification methods that span from classical machine learning algorithms to deep neural network models are provided. Gumpy can be used for both EEG and EMG biosignal analysis, visualization, real-time streaming and decoding. Results The usage of the toolbox was demonstrated through two different offline example studies, namely movement prediction from EEG motor imagery, and the decoding of natural grasp movements with the applied finger forces from surface EMG (sEMG) signals. Additionally, gumpy was used for real-time control of a robot arm using steady-state visually evoked potentials (SSVEP) as well as for real-time prosthetic hand control using sEMG. Overall, obtained results with the gumpy toolbox are comparable or better than previously reported results on the same datasets. Significance Gumpy is a free and open source software, which allows end-users to perform online hybrid BCIs and provides different techniques for processing and decoding of EEG and EMG signals. More importantly, the achieved results reveal that gumpy's deep learning toolbox can match or outperform the state-of-the-art in terms of accuracy. This can therefore enable BCI researchers to develop more robust decoding algorithms using novel techniques and hence chart a route ahead for new BCI improvements.

EVA1 is describing a new class of emotion-aware autonomous systems delivering intelligent personal assistant functionalities. EVA requires a multi-disciplinary approach, combining a number of critical building blocks into a cybernetics systems/software architecture: emotion aware systems and algorithms, multimodal interaction design, cognitive modelling, decision making and recommender systems, emotion sensing as feedback for learning, and distributed (edge) computing delivering cognitive services.

We present a mobile robot with sufficient computing power to simulate up to a quarter of a million neurons in real-time. We use this computing power, combined with various on-board sensory and motor systems (including silicon retinae) to implement a novel method for learning sensorimotor competences by example. That is, by temporarily manually controlling the robot, it can gather information about what sensorimotor mapping it should be performing. We show that such a learning-by-example system is well-suited to power efficient neuron-based computation (60?W for all quarter of a million neurons), that it can learn quickly (a few tens of seconds), and that its learning generalizes well to novel situations. We present a mobile omniwheel robot capable of running large-scale neural models.We demonstrate a software suite for running neural models on the robot.We demonstrate using this suite to learn complex parallel sensorimotor mappings.

All physical systems must reliably extract information from their noisy and partially observable environment and build an internal representation of space to orient their behaviour. Precise egomotion estimation is important to keep external (i.e. environmental information) and internal (i.e. proprioception) cues coherent. The constructed representation subsequently defines the space of possible actions. Due to the multimodal nature of incoming streams of sensory information, egomotion estimation is a challenging sensor fusion problem. In this paper we present a distributed cortically inspired processing scheme for sensor fusion, which given various sensory inputs, and simple relations defining inter-sensory dependencies, relaxes into a solution which provides a plausible interpretation of the perceived environment. The proposed model has been implemented for egomotion estimation on an autonomous mobile robot. We demonstrate that the model provides a precise estimate of both robot position and orientation. A cortically inspired sensor fusion network for robotic applications is introduced.Distributed graphical network, with combined feed-forward and recurrent connectivity.No global supervisor, only local processing, storage and exchange of information.Given sensory data, network relaxes into the best explanation of an underlying cause.Extensible to learn sensor correlations and adapt connectivity from incoming data.

Depth perception through stereo vision is an important feature of biological and artificial vision systems. While biological systems can compute disparities effortlessly, it requires intensive processing for artificial vision systems. The computing complexity resides in solving the correspondence problem – finding matching pairs of points in the two eyes. Inspired by the retina, event-based vision sensors allow a new constraint to solve the correspondence problem: time. Relying on precise spike-time, spiking neural networks can take advantage of this constraint. However, disparities can only be computed from dynamic environments since event-based vision sensors only report local changes in light intensity. In this paper, we show how microsaccadic eye movements can be used to compute disparities from static environments. To this end, we built a robotic head supporting two Dynamic Vision Sensors (DVS) capable of independent panning and simultaneous tilting. We evaluate the method on both static and dynamic scenes perceived through microsaccades. This paper demonstrates the complementarity of event-based vision sensors and active perception leading to more biologically inspired robots.

AlexNet is a Convolutional Neural Network (CNN) and reference in the field of Machine Learning for Deep Learning. It has been successfully applied to image classification, especially in large sets such as ImageNet. Here, we have successfully applied a smaller version of the AlexNet CNN to classify tropical fruits from the Supermarket Produce dataset. This database contains 2633 images of fruits divided into 15 categories with high variability and complexity, i.e. shadows, pose, occlusion, reflection (fruits inside a bag), etc. Since few training samples are required for fruit classification and to prevent overfitting, the modified AlexNet CNN has fewer feature maps and fully connected neurons than the original one, and data augmentation of the training set is used. Numerical results show a top-1 classification accuracy of 99.56 %, and a top-2 accuracy of 100 % for the 15 classes, which outperforms previous works on the same dataset.

In order to safely navigate and orient in their local surroundings autonomous systems need to rapidly extract and persistently track visual features from the environment. While there are many algorithms tackling those tasks for traditional frame-based cameras, these have to deal with the fact that conventional cameras sample their environment with a fixed frequency. Most prominently, the same features have to be found in consecutive frames and corresponding features then need to be matched using elaborate techniques as any information between the two frames is lost. We introduce a novel method to detect and track line structures in data streams of event-based silicon retinae [also known as dynamic vision sensors (DVS)]. In contrast to conventional cameras, these biologically inspired sensors generate a quasicontinuous stream of vision information analogous to the information stream created by the ganglion cells in mammal retinae. All pixels of DVS operate asynchronously without a periodic sampling rate and emit a so-called DVS address event as soon as they perceive a luminance change exceeding an adjustable threshold. We use the high temporal resolution achieved by the DVS to track features continuously through time instead of only at fixed points in time. The focus of this work lies on tracking lines in a mostly static environment which is observed by a moving camera, a typical setting in mobile robotics. Since DVS events are mostly generated at object boundaries and edges which in man-made environments often form lines they were chosen as feature to track. Our method is based on detecting planes of DVS address events in x-y-t-space and tracing these planes through time. It is robust against noise and runs in real time on a standard computer, hence it is suitable for low latency robotics. The efficacy and performance are evaluated on real-world data sets which show artificial structures in an office-building using event data for tracking and frame data for ground-truth estimation from a DAVIS240C sensor.

Non-invasive brain machine interfaces (BMIs) on motor imagery movements have been widely studied and used for many years to take advantage of the intuitive link between imagined motor tasks and natural actions. En route to future technical applications of neuromorphic computing, a major current challenge lies in the identification and implementation of brain inspired algorithms to decode recorded signals. Neuromorphic computing is believed to allow real-time implementation of large scale spiking models for processing and computation in non-invasive BMIs. Taking inspiration from the olfactory system of insects, we advance and implement a novel approach to decode and predict imaginary movements from electroencephalogram (EEG) signals. We use a spiking neural network implemented on SpiNNaker (4 chip, 64 cores) neuromorphic hardware. Our work provides a proof of concept for a successful implementation of a functional spiking neural network for decoding two motor imagery (MI) movements on the SpiNNaker system. The approach can be extended to classify more complex MI movements on larger SpiNNaker systems.

Ageing has an effect on many parameters of the physical condition, and one of them is the way a person walks. This property, the gait pattern, can unintrusively be observed by letting people walk over a sensor floor. The electric capacitance sensors built into the floor deliver information about when and where feet get into close proximity and contact with the floor during the phases of human locomotion. We processed gait patterns recorded this way by extracting a feature vector containing the discretised distribution of occurring geometrical extents of significant sensor readings. This kind of feature vector is an implicit measure encoding the ratio of swing-to stance phase timings in the gait cycle and representing how cleanly the leg swing is performed. We then used the dataset to train a Multi-Layer Perceptron to perform regression with the age of the person as the target value, and the feature vector as input. With this method and a dataset size of 142 persons recorded, we achieved a mean absolute error of approximately 10 years between the true age and the estimated age of the person. Considering the novelty of our approach, this is an acceptable result. The combination of a floor sensor and machine learning methods for interpreting the sensor data seems promising for further research and applications in care and medicine.

Instead of wastefully sending entire images at fixed frame rates, neuromorphic vision sensors only transmits the local pixel-level changes caused by movement in a scene at the time they occur. This results in a stream of events, with a latency in the order of micro-seconds. While these sensors offer tremendous advantages in terms of latency and bandwidth, they require new, adapted approaches to computer vision, due to their unique event-based pixel-level output. In this contribution, we propose an online multi-target tracking system utilizing for neuromorphic vision sensors, which is the first neuromorphic vision system in intelligent transportation systems. In order to track moving targets, a fast and simple object detection algorithm using clustering techniques is developed. To make full use of the low latency, we integrate an online tracking-by-clustering system running at a high frame rate, which far exceeds the real-time capabilities of traditional frame based industry cameras. The performance of the system is evaluated using real world dynamic vision sensor data of a highway bridge scenario. We hope that our attempt will motivate further research on neuromorphic vision sensors for intelligent transportation systems.

We demonstrate a spiking neural network that extracts spatial depth information from a stereoscopic visual input stream. The system makes use of a scalable neuromorphic computing platform, SpiNNaker, and neuromorphic vision sensors, so called silicon retinas, to solve the stereo matching (correspondence) problem in real-time. It dynamically fuses two retinal event streams into a depth-resolved event stream with a fixed latency of 2 ms, even at input rates as high as several 100,000 events per second. The network design is simple and portable so it can run on many types of neuromorphic computing platforms including FPGAs and dedicated silicon.

A factor graph (FG) can be considered as a unified model combining a Bayesian network (BN) and a Markov random field (MRF). The inference mechanism of a FG can be used to perform reasoning under incompleteness and uncertainty, which is a challenging task in many intelligent systems and robotics. Unfortunately, a complete inference mechanism requires intense computations that introduces a long delay for the reasoning process to complete. Furthermore, in an energy-constrained system such as a mobile robot, it is required to have a very efficient inference process. In this paper, we present an embedded FG inference engine that employs a neural-inspired discretization mechanism. The engine runs on a system-on-chip (SoC) and is accelerated by its FPGA. We optimized our design to balance the trade-off between speed and hardware resource utilization. In our fully-optimized design, it can accelerate the inference process eight times faster than the normal execution, which is twice the speed-up gain achieved by a parallelized FG running on a PC. The experiments demonstrate that our design can be extended into an efficient reconfigurable computing machine.

Proposed is a prototype of a wearable mobility device which aims to assist the blind with navigation and object avoidance via auditory-vision-substitution. The described system uses two dynamic vision sensors and event-based information processing techniques to extract depth information. The 3D visual input is then processed using three different strategies, and converted to a 3D output sound using an individualized head-related transfer function. The performance of the device with different processing strategies is evaluated via initial tests with ten subjects. The outcome of these tests demonstrate promising performance of the system after only very short training times of a few minutes due to the minimal encoding of outputs from the vision sensors which are translated into simple sound patterns easily interpretable for the user. The envisioned system will allow for efficient real-time algorithms on a hands-free and lightweight device with exceptional battery life-time.

We present an approach to estimate a person’s risk of falling by analysing gait recordings from a 6 m walk on a sensor floor. The risk of falls correlates with certain parameters of the human gait. Although these parameters are not measured directly with this sensor, their information is reflected in the data. The sensor floor works with a capacitance measurement principle and is sensitive to contact, such that persons standing, walking or lying on the floor are recognisable. In a preprocessing step, the person’s position is determined from the sensor data and the spread of contact points calculated. This spread implicitly contains the step sizes for the step phase in which two contact points are present. For each experiment, the distribution of occurring spreads is binned and taken as an input feature vector to a feed-forward perceptron. The neural network was trained by backpropagation with gait recordings from persons in low risk of falling and persons in high risk of falling. In the dataset, subjects were labelled as being in high risk of falling based on the prevalence of diseases, falls that already happened, and expert opinions. Though in this setup the data was recorded within a controlled environment, the results are transferable to larger installations and long-term observation periods.

We present a method for simultaneous localization and mapping for robots. The focus of our work is to use multiple small mobile robots which have only limited sensing and computational resources. Our robots each uses a laser pointer and an event based vision sensor to compute distances to its surroundings. The acquired data is used to update an occupancy map which can be shared among many robots at the same time. Here we demonstrate initial results for our proof-of-concept implementation. It runs in real-time using a mobile robot platform with an event based vision sensor for data acquisition. Distance estimates to objects are transferred to a remote computer to build a map of the environment. The results of our work can be used in future technical implementations, and for further investigations into cooperative mapping.

This paper presents a novel, drastically simplified method to compute optic flow on a miniaturized embedded vision system, suitable on-board of miniaturized indoor flying robots. Estimating optic flow is a common technique for robotic motion stabilization in systems without ground contact, such as unmanned aerial vehicles (UAVs). Because of high computing power requirements to process video camera data, most optic flow algorithms are implemented off-board on PCs or on dedicated hardware, connected through tethered or wireless links. Here, in contrast, we present a miniaturized stand-alone embedded system that utilizes a novel neuro-biologically inspired event-based vision sensor (DVS) to extract optic flow on-board in real-time with minimal computing requirements. The DVS provides asynchronous events that resemble temporal contrast changes at individual pixel level, instead of full image frames at regular time intervals. Such a representation provides high temporal resolution while simultaneously reducing the amount of data to be processed. We present a simple algorithm to extract optic flow information from such event-based vision data, which is sufficiently efficient in terms of data storage and processing power to be executed on an embedded 32bit ARM7 microcontroller in real-time. The developed stand-alone system is small, lightweight and energy efficient, and is ready to serve as sensor for ego motion estimates based on optic flow in autonomous UAVs.

This paper describes a robot head platform which holds a pair of event-based Dynamic Vision Sensor (DVS) retinas and microphones connected to an event-based binaural AEREAR2 VLSI cochlea system. The platform has 6 degrees of freedom (DOF): 2 for the neck, and 2 for each of the DVS retinas. Two applications using this platform are described: the first is image stitching of a scene larger than the field of view of the individual retinas as the head pans and tilts and the second is selective image painting of the local visual scene around spatially displaced sound sources. This platform allows for the investigation of event-driven sensory-action models that use the information from multiple event-based sensor modalities in real-time scenarios.

In order to perform appropriate actions, animals need to quickly and reliably classify their sensory input. How can representations suitable for classification be acquired from statistical properties of the animal’s natural environment? Akin to behavioural studies in rats, we investigate this question using texture discrimination by the vibrissae system as a model. To account for the rat’s active sensing behaviour, we record whisker movements in a hardware model. Based on these signals, we determine the response of primary neurons, modelled as spatio-temporal filters. Using their output, we train a second layer of neurons to optimise a temporal coherence objective function. The performance in classifying textures using a single cell strongly correlates with the cell’s temporal coherence; hence output cells outperform primary cells. Using a simple, unsupervised classifier, the performance on the output cell population is same as if using a sophisticated supervised classifier on the primary cells. Our results demonstrate that the optimisation of temporal coherence yields a representation that facilitates subsequent classification by selectively conveying relevant information.

Learning processes which take place during the development of a biological nervous system enable it to extract mappings between external stimuli and its internal state. Precise egomotion estimation is essential to keep these external and internal cues coherent given the rich multisensory environment. In this paper we present a learning model which, given various sensory inputs, converges to a state providing a coherent representation of the sensory space and the cross-sensory relations. The developed model, implemented for 3D egomotion estimation on a quadrotor, provides precise estimates for roll, pitch and yaw angles.

We developed a biologically plausible control algorithm to move the eyes of a six degrees of freedom robotic head in a human-like manner. Our neurocontroller, written with the neural simulator Nengo, integrates different biological neural models of eye movements, such as microsaccades, saccades, vestibular-ocular reflex, smooth pursuit and vergence. The coordination of the movements depends on the stream of sensory information acquired by two silicon retinas used as eyes and by an inertial measurement unit, which serves as a vestibular system. The eye movements generated by our neurocontroller resemble those of humans when exposed to the same visual input. This robotic platform can be used to investigate the most efficient exploration strategies used to extract salient features from either a static or dynamic visual scene. Future research should focus on technical enhancements and model refinements of the system.

The complexity of the kinematic and dynamic structure of humanoid robots make conventional analytical approaches to control increasingly unsuitable for such systems. Learning techniques offer a possible way to aid controller design if insufficient analytical knowledge is available, and learning approaches seem mandatory when humanoid systems are supposed to become completely autonomous. While recent research in neural networks and statistical learning has focused mostly on learning from finite data sets without stringent constraints on computational efficiency, learning for humanoid robots requires a different setting, characterized by the need for real-time learning performance from an essentially infinite stream of incrementally arriving data. This paper demonstrates how even high-dimensional learning problems of this kind can successfully be dealt with by techniques from nonparametric regression and locally weighted learning. As an example, we describe the application of one of the most advanced of such algorithms, Locally Weighted Projection Regression (LWPR), to the on-line learning of three problems in humanoid motor control: the learning of inverse dynamics models for model-based control, the learning of inverse kinematics of redundant manipulators, and the learning of oculomotor reflexes. All these examples demonstrate fast, i.e., within seconds or minutes, learning convergence with highly accurate final peformance. We conclude that real-time learning for complex motor system like humanoid robots is possible with appropriately tailored algorithms, such that increasingly autonomous robots with massive learning abilities should be achievable in the near future.

o We present the D-eDVSn a combined event-based 3D sensor n and a novel event-based full-3D simultaneous lo- calization and mapping algorithm which works exclusively with the sparse stream of visual data provided by the D-eDVS. The D-eDVS is a combination of the established PrimeSense RGB- D sensor and a biologically inspired embedded dynamic vision sensor. Dynamic vision sensors only react to dynamic contrast changes and output data in form of a sparse stream of events which represent individual pixel locations. We demonstrate how an event-based dynamic vision sensor can be fused with a classic frame-based RGB-D sensor to produce a sparse stream of depth-augmented 3D points. The advantages of a sparse, event-based stream are a much smaller amount of generated data, thus more efcient resource usage, and a continuous representation of motion allowing lag-free tracking. Our event- based SLAM algorithm is highly efcient and runs 20 times faster than realtime, provides localization updates at several hundred Hertz, and produces excellent results. We compare our method against ground truth from an external tracking system and two state-of-the-art algorithms on a new dataset which we release in combination with this paper.

Oculomotor control in a humanoid robot faces similar problems as biological oculomotor systems, that is, capturing targets accurately on a very narrow fovea, dealing with large delays in the control systems, the stabilization of gaze in the face of unknown perturbations of the body, selective attention, and the complexity of stereo vision. In this article, we suggest control circuits to realize three of the most basic oculomotor behaviors and their integration: the vestibulo-ocular reflex and optokinetic response (VOR–OKR) for gaze stabilization, smooth pursuit for tracking moving objects, and saccades for overt visual attention. Each of these behaviors and the mechanism for their integration were derived with inspiration from computational theories as well as behavioral and physiological data in neuroscience. Our implementations on humanoid demonstrate good performance of oculomotor behaviors, which proves to be a viable strategy to explore novel control mechanisms for humanoid robotics. Conversely, insights gained from our models have been able to directly influence views and provide new directions for computational neuroscience research.

One of the key computations performed in human brain is multi-sensory cue integration, through which humans are able to estimate the current state of the world to discover relative reliabilities and relations between observed cues. Mammalian cortex consists of highly distributed and interconnected populations of neurons, each providing a specific type of information about the state of the world. Connections between areas seemingly realize functional relationships amongst them and computation occurs by each area trying to be consistent with the areas it is connected to. In this paper using line-attraction dynamics and divisive normalization, we present a computational framework which is able to learn arbitrary non-linear relations between multiple cues using a simple Hebbian Learning principle. After learning, the network dynamics converges to the stable state so to satisfy the relation between connected populations. This network can perform several principle computational tasks such as inference, de-noising and cue-integration. By applying a real world multi-sensory integrating scenario, we demonstrate that the network can encode relative reliabilities of cues in different areas of the state space, over distributed population vectors. This reliability based encoding biases the network’s dynamics in favor of more reliable cues and realizes a near optimal sensory integration mechanism. Additional important features of the network are its scalability to cases with higher order of modalities and its flexibility to learn smooth functions of relations which is necessary for a system to operate in a dynamic environment.

We present a novel system for vision based anomaly detection in real-time environments. Our system uses an event-based vision sensor consisting of asynchronously operating pixels that is inspired by the human retina. Each pixel reports events of illumination changes, are processed in a purely event-based tracker that pursues edges of events in the input stream. The tracker estimates are used to determine whether the input events originate from anomalous or regular data. We distinguish between the two cases with a Growing Neural Gas (GNG), which is modified to suite our event-based processing pipeline. While learning of the GNG is supervised and performed offline, the detection is carried out online. We evaluate our system by inspection of fast-spinning cog-wheels. Our system achieves faster than real-time speed on commodity hardware and generalizes well to other cases. The results of this paper can be applied both to technical implementations where high speed but little processing power is required, and for further investigations into event-based algorithms.

Human mind from system perspective deals with high dimensional complex world as an adaptive Multi-Input Multi-Output complex system. This view is theorized by reductionism theory in philosophy of mind, where the world is represented as logical combination of simpler sub-systems for human so that operate with less energy. On the other hand, Human usually uses linguistic rules to describe and manipulate his expert knowledge about the world; the way that is well modeled by Fuzzy Logic. But how such a symbolic form of knowledge can be encoded and stored in plausible neural circuitry? Based on mentioned postulates, we have proposed an adaptive Neuro-Fuzzy machine in order to model a rule-based MIMO system as logical combination of spatially distributed Single-Input Single-Output sub-systems. Each SISO systems as sensory and processing layer of the inference system, construct a single rule and learning process is handled by a Hebbian-like Spike-Time Dependent Plasticity. To shape a concrete knowledge about the whole system, extracted features of SISO neural systems (or equivalently the rules associated with SISO systems) are combined. To exhibit the system applicability, a single link cart-pole balancer as a sensory-motor learning task, has been simulated. The system is provided by reinforcement feedback from environment and is able to learn how to get expert and achieve a successful policy to perform motor control.