Search

Abnormal event detection is an important task in research and industrial applications, which has received considerable attention in recent years. Existing methods usually rely on standard frame-based cameras to record the data and process them with computer vision technologies. In contrast, this paper presents a novel neuromorphic vision based abnormal event detection system. Compared to the frame-based camera, neuromorphic vision sensors, such as Dynamic Vision Sensor (DVS), do not acquire full images at a fixed frame rate but rather have independent pixels that output intensity changes (called events ) asynchronously at the time they occur. Thus, it avoids the design of the encryption scheme. Since events are triggered by moving edges on the scene, DVS is a natural motion detector for the abnormal objects and automatically filters out any temporally-redundant information. Based on this unique output, we first propose a highly efficient method based on the event density to select activated event cuboids and locate the foreground. We design a novel event -based multiscale spatio-temporal descriptor to extract features from the activated event cuboids for the abnormal event detection. Additionally, we build the NeuroAED dataset, the first public dataset dedicated to abnormal event detection with neuromorphic vision sensor. The NeuroAED dataset consists of four sub-datasets: Walking, Campus, Square, and Stair dataset. Experiments are conducted based on these datasets and demonstrate the high efficiency and accuracy of our method.

Event cameras are bio-inspired sensors that differ from conventional frame cameras: Instead of capturing images at a fixed rate, they asynchronously measure per-pixel brightness changes, and output a stream of events that encode the time, location and sign of the brightness changes. Event cameras offer attractive properties compared to traditional cameras: high temporal resolution (in the order of μ s), very high dynamic range (140 dB versus 60 dB), low power consumption, and high pixel bandwidth (on the order of kHz) resulting in reduced motion blur. Hence, event cameras have a large potential for robotics and computer vision in challenging scenarios for traditional cameras, such as low-latency, high speed, and high dynamic range. However, novel methods are required to process the unconventional output of these sensors in order to unlock their potential. This paper provides a comprehensive overview of the emerging field of event-based vision, with a focus on the applications and the algorithms developed to unlock the outstanding properties of event cameras. We present event cameras from their working principle, the actual sensors that are available and the tasks that they have been used for, from low-level vision (feature detection and tracking, optic flow, etc.) to high-level vision (reconstruction, segmentation, recognition). We also discuss the techniques developed to process events, including learning-based techniques, as well as specialized processors for these novel sensors, such as spiking neural networks. Additionally, we highlight the challenges that remain to be tackled and the opportunities that lie ahead in the search for a more efficient, bio-inspired way for machines to perceive and interact with the world., IEEE Transactions on Pattern Analysis and Machine Intelligence, 44 (1), ISSN:0162-8828, ISSN:1939-3539

Neurorobotic mimics the structural and functional principles of living creature systems. Modeling a single system by robotic hardware and software has existed for decades. However, an integrated toolset studying the interaction of all systems has not been demonstrated yet. We present a hybrid neuromorphic computing paradigm to bridge this gap by combining the neurorobotics platform (NRP) with the neuromorphic snake-like robot (NeuroSnake). This paradigm encompasses the virtual models, neuromorphic sensing and computing capabilities, and physical bio-inspired bodies, with which an experimenter can design and execute both in-silico and in-vivo robotic experimentation easily. The NRP is a public Web-based platform for easily testing brain models with virtual bodies and environments. The NeuroSnake is a bio-inspired robot equipped with a silico-retina sensor and neuromorphic computer for power-efficiency applications. We illustrate the efficiencies of our paradigm with an easy designing of a visual pursuit experiment in the NRP. We study two automatic behavior learning tasks which are further integrated into a complex task of semi-autonomous pole climbing. The result shows that robots could build new learning rules in a less explicit manner inspired by living creatures. Our method gives an alternative way to efficiently develop complex behavior control of the ro As spiking neural network is a bio-inspired neural network and the NeuroSnake robot is equipped with a spike-based silicon retina camera, the control system can be easily implemented via spiking neurons simulated on neuromorphic hardware, such as SpiNNaker.bot.

The cameras in modern gaze-tracking systems suffer from fundamental bandwidth and power limitations, constraining data acquisition speed to 300 Hz realistically. This obstructs the use of mobile eye trackers to perform, e.g., low latency predictive rendering, or to study quick and subtle eye motions like microsaccades using head-mounted devices in the wild. Here, we propose a hybrid frame-event-based near-eye gaze tracking system offering update rates beyond 10,000 Hz with an accuracy that matches that of high-end desktop-mounted commercial trackers when evaluated in the same conditions. Our system, previewed in Figure 1, builds on emerging event cameras that simultaneously acquire regularly sampled frames and adaptively sampled events. We develop an online 2D pupil fitting method that updates a parametric model every one or few events. Moreover, we propose a polynomial regressor for estimating the point of gaze from the parametric pupil model in real time. Using the first event-based gaze dataset, we demonstrate that our system achieves accuracies of 0.45°-1.75° for fields of view from 45° to 98°. With this technology, we hope to enable a new generation of ultra-low-latency gaze-contingent rendering and display techniques for virtual and augmented reality.

Restoring tactile sensation is essential to enable in-hand manipulation and the smooth, natural control of upper-limb prosthetic devices. Here we present a platform to contribute to that long-term vision, combining an anthropomorphic robot hand (QB SoftHand) with a neuromorphic optical tactile sensor (neuroTac). Neuromorphic sensors aim to produce efficient, spike-based representations of information for bio-inspired processing. The development of this 5-fingered, sensorized hardware platform is validated with a customized mount allowing manual control of the hand. The platform is demonstrated to succesfully identify 4 objects from the YCB object set, and accurately discriminate between 4 directions of shear during stable grasps. This platform could lead to wide-ranging developments in the areas of haptics, prosthetics and telerobotics.

We propose a neuromorphic approach to perception, reasoning and motor control using Spiking Neural Networks in mobile robotics. We demonstrate this by using a mobile robotic manipulator solving a pick-and-place task. All sensory data is provided by spike-based silicon retina cameras - eDVS (embedded Dynamic Vision Sensor) - and all reasoning and motor control is implemented in Spiking Neural Networks. For the given scenario, the robot is capable of detecting a sequence of objects blinking at different frequencies, finding one object that is not in the right place of the sequence, picking up this object and moving it to its correct position. Such a scenario demonstrates how to build large-scale networks solving a high-level cognitive task by combining several smaller networks responsible for low-level tasks. Importantly, here we focus only on generating a neural network that is capable of performing the task. This will be the basis of future work using neural network learning algorithms to improve task performance. The long-term goal is to learn sophisticated behaviours by experience while at the same time being able to introduce expert knowledge for intermediate tasks that can be used to initialize the network or to speed up the learning process.

Factor graphs are probabilistic graphical frameworks for modeling complex and dynamic systems. They can be used in a broad range of application domains, from machine learning and robotics, to signal processing and digital communications. One important aspect that makes a factor graph very useful and very promising to be applied widely is its inference mechanism that is suitable for performing a complex model-based reasoning. However, its features have not fully explored and factor graphs are still used mainly as modeling tools that run on standard computers. Whereas in real applications such as robotics, one needs a practical implementation of such a framework. In this paper, we describe the development of a factor-graph-based inference engine that runs on a System-on-Chip (SoC). Running natively on a low level hardware, our factor graph engine delivers highest performance for real-time applications. We designed the embedded architecture so that it conveys important aspects such as modularity, scalability, flexibility and platform-friendly framework. The proposed architecture has customizable levels of parallelism as well as re-configurable modules that are extensible to accommodate large networks. We optimized the design to achieve high efficiency in terms of clock latency and resources consumption. We have tested our design on Xilinx Zynq-7000 SoCs and the implementation result demonstrates that the proposed framework can potentially be extended into a massively distributed probabilistic computing engine.

Predicting future behavior of other traffic participants is an essential task that needs to be solved by automated vehicles and human drivers alike to achieve safe and situation-aware driving. Modern approaches to vehicles trajectory prediction typically rely on data-driven models like neural networks, in particular LSTMs (Long Short-Term Memorys), achieving promising results. However, the question of optimal composition of the underlying training data has received less attention. In this paper, we expand on previous work on vehicle trajectory prediction based on neural network models employing distributed representations to encode automotive scenes in a semantic vector substrate. We analyze the influence of variations in the training data on the performance of our prediction models. Thereby, we show that the models employing our semantic vector representation outperform the numerical model when trained on an adequate data set and thereby, that the composition of training data in vehicle trajectory prediction is crucial for successful training. We conduct our analysis on challenging real-world driving data.

With the advanced development of image processing technology, visible light positioning (VLP) system based on image sensors has attracted more and more attention. However, as a commonly used light receiver, traditional CMOS camera has limited dynamic range and high latency, which is susceptible to various lighting and environmental factors. Moreover, high computational cost from image processing is unavoidable for most of visible light positioning systems. In our work, a novel VLP system using an event-based neuromorphic vision sensor (event camera) as the light receiver is proposed. Due to the low latency and microsecond-level temporal resolution of the event camera, our VLP system is able to identify multiple high-frequency flickering LEDs in asynchronous events simultaneously leaving out the need for data association and traditional image processing methods. A multi-LED fusion method is applied and a high positioning accuracy of 3cm is achieved when the height between LEDs and the event camera is within 1m.

Drowsiness driving is a principal factor of many fatal traffic accidents. This paper presents the first event-based drowsiness driving detection (EDDD) system by using the recently developed neuromorphic vision sensor. Compared with traditional frame-based cameras, neuromorphic vision sensors, such as Dynamic Vision Sensors (DVS), have a high dynamic range and do not acquire full images at a fixed frame rate but rather have independent pixels that output intensity changes (called events) asynchronously at the time they occur. Since events are generated by moving edges in the scene, DVS is considered as an efficient and effective detector for the drowsiness driving-related motions. Based on this unique output, this work first proposes a highly efficient method to recognize and localize the driver’s eyes and mouth motions from event streams. We further design and extract event-based drowsiness-related features directly from the event streams caused by eyes and mouths motions, then the EDDD model is established based on these features. Additionally, we provide the EDDD dataset, the first public dataset dedicated to event-based drowsiness driving detection. The EDDD dataset has 260 recordings in daytime and evening with several challenging scenes such as subjects wearing glasses/sunglasses. Experiments are conducted based on this dataset and demonstrate the high efficiency and accuracy of our method under different illumination conditions. As the first investigation of the usage of DVS in drowsiness driving detection applications, we hope that this work will inspire more event-based drowsiness driving detection research.

Predicting future behavior and positions of other traffic participants from observations is a key problem that needs to be solved by human drivers and automated vehicles alike to safely navigate their environment and to reach their desired goal. In this paper, we expand on previous work on an automotive environment model based on vector symbolic architectures (VSAs). We investigate a vector-representation to encapsulate spatial information of multiple objects based on a convolutive power encoding. Assuming that future positions of vehicles are influenced not only by their own past positions and dynamics (e.g., velocity and acceleration) but also by the behavior of the other traffic participants in the vehicle's surroundings, our motivation is 3-fold: we hypothesize that our structured vector-representation will be able to capture these relations and mutual influence between multiple traffic participants. Furthermore, the dimension of the encoding vectors remains fixed while being independent of the number of other vehicles encoded in addition to the target vehicle. Finally, a VSA-based encoding allows us to combine symbol-like processing with the advantages of neural network learning. In this work, we use our vector representation as input for a long short-term memory (LSTM) network for sequence to sequence prediction of vehicle positions. In an extensive evaluation, we compare this approach to other LSTM-based benchmark systems using alternative data encoding schemes, simple feed-forward neural networks as well as a simple linear prediction model for reference. We analyze advantages and drawbacks of the presented methods and identify specific driving situations where our approach performs best. We use characteristics specifying such situations as a foundation for an online-learning mixture-of-experts prototype, which chooses at run time between several available predictors depending on the current driving situation to achieve the best possible forecast.

In this paper we describe the NeuroRobotic Mouse (NeRmo) a low-cost, modular bio mimetic robot, mimicking the actuation and walking behaviour of a common mouse (Mus musculus). This latest version has 13 Degrees of Freedom with 21 tendon driven joints and can be controlled in both open and closed loop. It is capable of different gaits as well as keeping the body upright when in a sitting position. The robot includes joint position sensors, pressure sensors on the soles of the feet as well as two cameras in the head. As the design of this robotic platform was inspired by detailed observations of the biomechanics of mice and rats, it can be used in motion research using animal data as am element of comparison, or even as actuation input.

Anthropomimetic robots sense, behave, interact, and feel like humans. By this definition, they require human-like physical hardware and actuation but also brain-like control and sensing. The most self-evident realization to meet those requirements would be a human-like musculoskeletal robot with a brain-like neural controller. While both musculoskeletal robotic hardware and neural control software have existed for decades, a scalable approach that could be used to build and control an anthropomimetic human-scale robot has not yet been demonstrated. Combining Myorobotics, a framework for musculoskeletal robot development, with SpiNNaker, a neuromorphic computing platform, we present the proof of principle of a system that can scale to dozens of neurally controlled, physically compliant joints. At its core, it implements a closed-loop cerebellar model that provides real-time, low-level, neural control at minimal power consumption and maximal extensibility. Higher-order (e.g., cortical) neural networks and neuromorphic sensors like silicon retinae or cochleae can be incorporated.

Predicting future motion of other vehicles or, more generally, the development of traffic situations, is an essential step towards secure, context-aware automated driving. On the one hand, human drivers are able to anticipate driving situations continuously based on the currently perceived behavior of other traffic participants while incorporating prior experience. On the other hand, the most successful data-driven prediction models are typically trained on large amounts of recorded data before deployment achieving remarkable results. In this paper, we present a mixture-of-experts online learning model encapsulating both ideas. Our system learns at run time to choose between several models, which have been previously trained offline, based on the current situational context. We show that our model is able to improve over the offline models already after a short ramp-up phase. We evaluate our system on real world driving data.

Objective. The objective of this work is to use the capability of spiking neural networks to capture the spatio-temporal information encoded in time-series signals and decode them without the use of hand-crafted features and vector-based learning and the realization of the spiking model on low-power neuromorphic hardware. Approach. The NeuCube spiking model was used to classify different grasp movements directly from raw surface electromyography signals (sEMG), the estimations of the applied finger forces as well as the classification of two motor imagery movements from raw electroencephalography (EEG). In a parallel investigation, the designed spiking decoder was implemented on SpiNNaker neuromorphic hardware, which allows low-energy real-time processing. Main results. Experimental results reveal a better classification accuracy using the NeuCube model compared to traditionalmachine learning methods. For sEMG classification, we reached a training accuracy of 85% and a test accuracy of 84.8%, as well as less than 19% of relative root mean square error (rRMSE) when estimating finger forces from six subjects. For the EEG classification, a mean accuracy of 75% was obtained when tested on raw EEG data from nine subjects from the existing 2b dataset from ‘BCI competition IV’. Significance. This work provides a proof of concept for a successful implementation of the NeuCube spiking model on the SpiNNakerneuromorphic platform for raw sEMG and EEG decoding, which could chart a route ahead for a new generation of portable closed-loop and low-power neuroprostheses.

Non-invasive, electroencephalography (EEG)-based brain-computer interfaces (BCIs) on motor imagery movements translate the subject’s motor intention into control signals through classifying the EEG patterns caused by different imagination tasks, e.g., hand movements. This type of BCI has been widely studied and used as an alternative mode of communication and environmental control for disabled patients, such as those suffering from a brainstem stroke or a spinal cord injury (SCI). Notwithstanding the success of traditional machine learning methods in classifying EEG signals, these methods still rely on hand-crafted features. The extraction of such features is a difficult task due to the high non-stationarity of EEG signals, which is a major cause by the stagnating progress in classification performance. Remarkable advances in deep learning methods allow end-to-end learning without any feature engineering, which could benefit BCI motor imagery applications. We developed three deep learning models: (1) A long short-term memory (LSTM), (2) a spectrogram-based convolutional neural network model (CNN), and (3) a recurrent convolutional neural network (RCNN), for decoding motor imagery movements directly from raw EEG signals without (any manual) feature engineering. Results were evaluated on our own publicly available, EEG data collected from 20 subjects and on an existing dataset known as 2b EEG dataset from “BCI Competition IV”. Overall, better classification performance was achieved with deep learning models compared to state-of-the art machine learning techniques, which could chart a route ahead for developing new robust techniques for EEG signal decoding. We underpin this point by demonstrating the successful real-time control of a robotic arm using our CNN based BCI.

Objective The objective of this work is to present gumpy, a new free and open source Python toolbox designed for hybrid brain-computer interface (BCI). Approach Gumpy provides state-of-the-art algorithms and includes a rich selection of signal processing methods that have been employed by the BCI community over the last 20 years. In addition, a wide range of classification methods that span from classical machine learning algorithms to deep neural network models are provided. Gumpy can be used for both EEG and EMG biosignal analysis, visualization, real-time streaming and decoding. Results The usage of the toolbox was demonstrated through two different offline example studies, namely movement prediction from EEG motor imagery, and the decoding of natural grasp movements with the applied finger forces from surface EMG (sEMG) signals. Additionally, gumpy was used for real-time control of a robot arm using steady-state visually evoked potentials (SSVEP) as well as for real-time prosthetic hand control using sEMG. Overall, obtained results with the gumpy toolbox are comparable or better than previously reported results on the same datasets. Significance Gumpy is a free and open source software, which allows end-users to perform online hybrid BCIs and provides different techniques for processing and decoding of EEG and EMG signals. More importantly, the achieved results reveal that gumpy's deep learning toolbox can match or outperform the state-of-the-art in terms of accuracy. This can therefore enable BCI researchers to develop more robust decoding algorithms using novel techniques and hence chart a route ahead for new BCI improvements.

Simultaneous localization and mapping (SLAM) is one of the core tasks of mobile autonomous robots. Looking for power efficient and embedded solutions for SLAM is an important challenge when building controllers for small and agile robots. Biological neural systems of even simple animals are until now unprecedented in their ability to localize themselves in an unknown environment. Neuromorphic engineering offers ultra low-power and compact computing hardware, in which biologically inspired neuronal architectures for SLAM can be realised. In this paper, we propose an on chip approach for one of the components of SLAM: path integration. Our solution takes inspiration from biology and uses motor command information to estimate the orientation of an agent solely in a spiking neural network. We realise this network on a neuromorphic device that implements artificial neurons and synapses with analog electronics. The neural network receives visual input from an event-based camera and uses this information to correct the on-chip spiking neurons estimate of the robot's orientation. This system can be easily integrated with other localization and mapping components on chip and is a step towards a fully neuromorphic SLAM.

The problem of inferring distances from a visual sensor to objects in a scene — referred to as depth estimation — can be solved in various ways. Among those, stereo vision is a method in which two sensors observe the same scene from different viewpoints. To recover the three-dimensional coordinates of a point, its two projections — one in each view — can be used for triangulation. However, the pair of points in the two views that correspond to each other has to be found first. This is known as stereo-matching and is usually a computationally expensive operation. Traditionally, this is performed by describing a point in the first view with some information from its surrounding, e.g. in a feature vector, and then searching for a match with a point described in a similar way in the other view. In this work, we propose a simple idea that alleviates this stereo-matching problem using an active component: a mirror-galvanometer driven laser. The laser beam is deflected by actuating two mirrors, thus creating a sequence of "light spots" in the scene. At these spots, contrast changes quickly. We capture those contrast changes by two Dynamic Vision Sensors (DVS). The high time-resolution of these sensors enables the detection of the laser-induced events in time and their matching using lightweight computation. This method enables event-based depth estimation at a high speed, low computational cost, and without exact sensor synchronization.

We present a mobile robot with sufficient computing power to simulate up to a quarter of a million neurons in real-time. We use this computing power, combined with various on-board sensory and motor systems (including silicon retinae) to implement a novel method for learning sensorimotor competences by example. That is, by temporarily manually controlling the robot, it can gather information about what sensorimotor mapping it should be performing. We show that such a learning-by-example system is well-suited to power efficient neuron-based computation (60?W for all quarter of a million neurons), that it can learn quickly (a few tens of seconds), and that its learning generalizes well to novel situations. We present a mobile omniwheel robot capable of running large-scale neural models.We demonstrate a software suite for running neural models on the robot.We demonstrate using this suite to learn complex parallel sensorimotor mappings.

All physical systems must reliably extract information from their noisy and partially observable environment and build an internal representation of space to orient their behaviour. Precise egomotion estimation is important to keep external (i.e. environmental information) and internal (i.e. proprioception) cues coherent. The constructed representation subsequently defines the space of possible actions. Due to the multimodal nature of incoming streams of sensory information, egomotion estimation is a challenging sensor fusion problem. In this paper we present a distributed cortically inspired processing scheme for sensor fusion, which given various sensory inputs, and simple relations defining inter-sensory dependencies, relaxes into a solution which provides a plausible interpretation of the perceived environment. The proposed model has been implemented for egomotion estimation on an autonomous mobile robot. We demonstrate that the model provides a precise estimate of both robot position and orientation. A cortically inspired sensor fusion network for robotic applications is introduced.Distributed graphical network, with combined feed-forward and recurrent connectivity.No global supervisor, only local processing, storage and exchange of information.Given sensory data, network relaxes into the best explanation of an underlying cause.Extensible to learn sensor correlations and adapt connectivity from incoming data.

Depth perception through stereo vision is an important feature of biological and artificial vision systems. While biological systems can compute disparities effortlessly, it requires intensive processing for artificial vision systems. The computing complexity resides in solving the correspondence problem – finding matching pairs of points in the two eyes. Inspired by the retina, event-based vision sensors allow a new constraint to solve the correspondence problem: time. Relying on precise spike-time, spiking neural networks can take advantage of this constraint. However, disparities can only be computed from dynamic environments since event-based vision sensors only report local changes in light intensity. In this paper, we show how microsaccadic eye movements can be used to compute disparities from static environments. To this end, we built a robotic head supporting two Dynamic Vision Sensors (DVS) capable of independent panning and simultaneous tilting. We evaluate the method on both static and dynamic scenes perceived through microsaccades. This paper demonstrates the complementarity of event-based vision sensors and active perception leading to more biologically inspired robots.

AlexNet is a Convolutional Neural Network (CNN) and reference in the field of Machine Learning for Deep Learning. It has been successfully applied to image classification, especially in large sets such as ImageNet. Here, we have successfully applied a smaller version of the AlexNet CNN to classify tropical fruits from the Supermarket Produce dataset. This database contains 2633 images of fruits divided into 15 categories with high variability and complexity, i.e. shadows, pose, occlusion, reflection (fruits inside a bag), etc. Since few training samples are required for fruit classification and to prevent overfitting, the modified AlexNet CNN has fewer feature maps and fully connected neurons than the original one, and data augmentation of the training set is used. Numerical results show a top-1 classification accuracy of 99.56 %, and a top-2 accuracy of 100 % for the 15 classes, which outperforms previous works on the same dataset.

Neuromorphic electronic systems exhibit advantageous characteristics, in terms of low energy consumption and low response latency, which can be useful in robotic applications that require compact and low power embedded computing resources. However, these neuromorphic circuits still face significant limitations that make their usage challenging: these include low precision, variability of components, sensitivity to noise and temperature drifts, as well as the currently limited number of neurons and synapses that are typically emulated on a single chip. In this paper, we show how it is possible to achieve functional robot control strategies using a mixed signal analog/digital neuromorphic processor interfaced to a mobile robotic platform equipped with an event-based dynamic vision sensor. We provide a proof of concept implementation of obstacle avoidance and target acquisition using biologically plausible spiking neural networks directly emulated by the neuromorphic hardware. To our knowledge, this is the first demonstration of a working spike-based neuromorphic robotic controller in this type of hardware which illustrates the feasibility, as well as limitations, of this approach.

Neuromorphic hardware emulates dynamics of biological neural networks in electronic circuits offering an alternative to the von Neumann computing architecture that is low-power, inherently parallel, and event-driven. This hardware allows to implement neural-network based robotic controllers in an energy-efficient way with low latency, but requires solving the problem of device variability, characteristic for analog electronic circuits. In this work, we interfaced a mixed-signal analog-digital neuromorphic processor ROLLS to a neuromorphic dynamic vision sensor (DVS) mounted on a robotic vehicle and developed an autonomous neuromorphic agent that is able to perform neurally inspired obstacle-avoidance and target acquisition. We developed a neural network architecture that can cope with device variability and verified its robustness in different environmental situations, e.g., moving obstacles, moving target, clutter, and poor light conditions. We demonstrate how this network, combined with the properties of the DVS, allows the robot to avoid obstacles using a simple biologically-inspired dynamics. We also show how a Dynamic Neural Field for target acquisition can be implemented in spiking neuromorphic hardware. This work demonstrates an implementation of working obstacle avoidance and target acquisition using mixed signal analog/digital neuromorphic hardware., Frontiers in Neurorobotics, 11, ISSN:1011-5218

Non-invasive brain machine interfaces (BMIs) on motor imagery movements have been widely studied and used for many years to take advantage of the intuitive link between imagined motor tasks and natural actions. En route to future technical applications of neuromorphic computing, a major current challenge lies in the identification and implementation of brain inspired algorithms to decode recorded signals. Neuromorphic computing is believed to allow real-time implementation of large scale spiking models for processing and computation in non-invasive BMIs. Taking inspiration from the olfactory system of insects, we advance and implement a novel approach to decode and predict imaginary movements from electroencephalogram (EEG) signals. We use a spiking neural network implemented on SpiNNaker (4 chip, 64 cores) neuromorphic hardware. Our work provides a proof of concept for a successful implementation of a functional spiking neural network for decoding two motor imagery (MI) movements on the SpiNNaker system. The approach can be extended to classify more complex MI movements on larger SpiNNaker systems.

Ageing has an effect on many parameters of the physical condition, and one of them is the way a person walks. This property, the gait pattern, can unintrusively be observed by letting people walk over a sensor floor. The electric capacitance sensors built into the floor deliver information about when and where feet get into close proximity and contact with the floor during the phases of human locomotion. We processed gait patterns recorded this way by extracting a feature vector containing the discretised distribution of occurring geometrical extents of significant sensor readings. This kind of feature vector is an implicit measure encoding the ratio of swing-to stance phase timings in the gait cycle and representing how cleanly the leg swing is performed. We then used the dataset to train a Multi-Layer Perceptron to perform regression with the age of the person as the target value, and the feature vector as input. With this method and a dataset size of 142 persons recorded, we achieved a mean absolute error of approximately 10 years between the true age and the estimated age of the person. Considering the novelty of our approach, this is an acceptable result. The combination of a floor sensor and machine learning methods for interpreting the sensor data seems promising for further research and applications in care and medicine.

Instead of wastefully sending entire images at fixed frame rates, neuromorphic vision sensors only transmits the local pixel-level changes caused by movement in a scene at the time they occur. This results in a stream of events, with a latency in the order of micro-seconds. While these sensors offer tremendous advantages in terms of latency and bandwidth, they require new, adapted approaches to computer vision, due to their unique event-based pixel-level output. In this contribution, we propose an online multi-target tracking system utilizing for neuromorphic vision sensors, which is the first neuromorphic vision system in intelligent transportation systems. In order to track moving targets, a fast and simple object detection algorithm using clustering techniques is developed. To make full use of the low latency, we integrate an online tracking-by-clustering system running at a high frame rate, which far exceeds the real-time capabilities of traditional frame based industry cameras. The performance of the system is evaluated using real world dynamic vision sensor data of a highway bridge scenario. We hope that our attempt will motivate further research on neuromorphic vision sensors for intelligent transportation systems.

Proposed is a prototype of a wearable mobility device which aims to assist the blind with navigation and object avoidance via auditory-vision-substitution. The described system uses two dynamic vision sensors and event-based information processing techniques to extract depth information. The 3D visual input is then processed using three different strategies, and converted to a 3D output sound using an individualized head-related transfer function. The performance of the device with different processing strategies is evaluated via initial tests with ten subjects. The outcome of these tests demonstrate promising performance of the system after only very short training times of a few minutes due to the minimal encoding of outputs from the vision sensors which are translated into simple sound patterns easily interpretable for the user. The envisioned system will allow for efficient real-time algorithms on a hands-free and lightweight device with exceptional battery life-time.

We present an approach to estimate a person’s risk of falling by analysing gait recordings from a 6 m walk on a sensor floor. The risk of falls correlates with certain parameters of the human gait. Although these parameters are not measured directly with this sensor, their information is reflected in the data. The sensor floor works with a capacitance measurement principle and is sensitive to contact, such that persons standing, walking or lying on the floor are recognisable. In a preprocessing step, the person’s position is determined from the sensor data and the spread of contact points calculated. This spread implicitly contains the step sizes for the step phase in which two contact points are present. For each experiment, the distribution of occurring spreads is binned and taken as an input feature vector to a feed-forward perceptron. The neural network was trained by backpropagation with gait recordings from persons in low risk of falling and persons in high risk of falling. In the dataset, subjects were labelled as being in high risk of falling based on the prevalence of diseases, falls that already happened, and expert opinions. Though in this setup the data was recorded within a controlled environment, the results are transferable to larger installations and long-term observation periods.

We present a method for simultaneous localization and mapping for robots. The focus of our work is to use multiple small mobile robots which have only limited sensing and computational resources. Our robots each uses a laser pointer and an event based vision sensor to compute distances to its surroundings. The acquired data is used to update an occupancy map which can be shared among many robots at the same time. Here we demonstrate initial results for our proof-of-concept implementation. It runs in real-time using a mobile robot platform with an event based vision sensor for data acquisition. Distance estimates to objects are transferred to a remote computer to build a map of the environment. The results of our work can be used in future technical implementations, and for further investigations into cooperative mapping.

This paper presents a novel, drastically simplified method to compute optic flow on a miniaturized embedded vision system, suitable on-board of miniaturized indoor flying robots. Estimating optic flow is a common technique for robotic motion stabilization in systems without ground contact, such as unmanned aerial vehicles (UAVs). Because of high computing power requirements to process video camera data, most optic flow algorithms are implemented off-board on PCs or on dedicated hardware, connected through tethered or wireless links. Here, in contrast, we present a miniaturized stand-alone embedded system that utilizes a novel neuro-biologically inspired event-based vision sensor (DVS) to extract optic flow on-board in real-time with minimal computing requirements. The DVS provides asynchronous events that resemble temporal contrast changes at individual pixel level, instead of full image frames at regular time intervals. Such a representation provides high temporal resolution while simultaneously reducing the amount of data to be processed. We present a simple algorithm to extract optic flow information from such event-based vision data, which is sufficiently efficient in terms of data storage and processing power to be executed on an embedded 32bit ARM7 microcontroller in real-time. The developed stand-alone system is small, lightweight and energy efficient, and is ready to serve as sensor for ego motion estimates based on optic flow in autonomous UAVs.

This paper describes a robot head platform which holds a pair of event-based Dynamic Vision Sensor (DVS) retinas and microphones connected to an event-based binaural AEREAR2 VLSI cochlea system. The platform has 6 degrees of freedom (DOF): 2 for the neck, and 2 for each of the DVS retinas. Two applications using this platform are described: the first is image stitching of a scene larger than the field of view of the individual retinas as the head pans and tilts and the second is selective image painting of the local visual scene around spatially displaced sound sources. This platform allows for the investigation of event-driven sensory-action models that use the information from multiple event-based sensor modalities in real-time scenarios.

In order to perform appropriate actions, animals need to quickly and reliably classify their sensory input. How can representations suitable for classification be acquired from statistical properties of the animal’s natural environment? Akin to behavioural studies in rats, we investigate this question using texture discrimination by the vibrissae system as a model. To account for the rat’s active sensing behaviour, we record whisker movements in a hardware model. Based on these signals, we determine the response of primary neurons, modelled as spatio-temporal filters. Using their output, we train a second layer of neurons to optimise a temporal coherence objective function. The performance in classifying textures using a single cell strongly correlates with the cell’s temporal coherence; hence output cells outperform primary cells. Using a simple, unsupervised classifier, the performance on the output cell population is same as if using a sophisticated supervised classifier on the primary cells. Our results demonstrate that the optimisation of temporal coherence yields a representation that facilitates subsequent classification by selectively conveying relevant information.

Learning processes which take place during the development of a biological nervous system enable it to extract mappings between external stimuli and its internal state. Precise egomotion estimation is essential to keep these external and internal cues coherent given the rich multisensory environment. In this paper we present a learning model which, given various sensory inputs, converges to a state providing a coherent representation of the sensory space and the cross-sensory relations. The developed model, implemented for 3D egomotion estimation on a quadrotor, provides precise estimates for roll, pitch and yaw angles.

We developed a biologically plausible control algorithm to move the eyes of a six degrees of freedom robotic head in a human-like manner. Our neurocontroller, written with the neural simulator Nengo, integrates different biological neural models of eye movements, such as microsaccades, saccades, vestibular-ocular reflex, smooth pursuit and vergence. The coordination of the movements depends on the stream of sensory information acquired by two silicon retinas used as eyes and by an inertial measurement unit, which serves as a vestibular system. The eye movements generated by our neurocontroller resemble those of humans when exposed to the same visual input. This robotic platform can be used to investigate the most efficient exploration strategies used to extract salient features from either a static or dynamic visual scene. Future research should focus on technical enhancements and model refinements of the system.

The complexity of the kinematic and dynamic structure of humanoid robots make conventional analytical approaches to control increasingly unsuitable for such systems. Learning techniques offer a possible way to aid controller design if insufficient analytical knowledge is available, and learning approaches seem mandatory when humanoid systems are supposed to become completely autonomous. While recent research in neural networks and statistical learning has focused mostly on learning from finite data sets without stringent constraints on computational efficiency, learning for humanoid robots requires a different setting, characterized by the need for real-time learning performance from an essentially infinite stream of incrementally arriving data. This paper demonstrates how even high-dimensional learning problems of this kind can successfully be dealt with by techniques from nonparametric regression and locally weighted learning. As an example, we describe the application of one of the most advanced of such algorithms, Locally Weighted Projection Regression (LWPR), to the on-line learning of three problems in humanoid motor control: the learning of inverse dynamics models for model-based control, the learning of inverse kinematics of redundant manipulators, and the learning of oculomotor reflexes. All these examples demonstrate fast, i.e., within seconds or minutes, learning convergence with highly accurate final peformance. We conclude that real-time learning for complex motor system like humanoid robots is possible with appropriately tailored algorithms, such that increasingly autonomous robots with massive learning abilities should be achievable in the near future.

o We present the D-eDVSn a combined event-based 3D sensor n and a novel event-based full-3D simultaneous lo- calization and mapping algorithm which works exclusively with the sparse stream of visual data provided by the D-eDVS. The D-eDVS is a combination of the established PrimeSense RGB- D sensor and a biologically inspired embedded dynamic vision sensor. Dynamic vision sensors only react to dynamic contrast changes and output data in form of a sparse stream of events which represent individual pixel locations. We demonstrate how an event-based dynamic vision sensor can be fused with a classic frame-based RGB-D sensor to produce a sparse stream of depth-augmented 3D points. The advantages of a sparse, event-based stream are a much smaller amount of generated data, thus more efcient resource usage, and a continuous representation of motion allowing lag-free tracking. Our event- based SLAM algorithm is highly efcient and runs 20 times faster than realtime, provides localization updates at several hundred Hertz, and produces excellent results. We compare our method against ground truth from an external tracking system and two state-of-the-art algorithms on a new dataset which we release in combination with this paper.

Oculomotor control in a humanoid robot faces similar problems as biological oculomotor systems, that is, capturing targets accurately on a very narrow fovea, dealing with large delays in the control systems, the stabilization of gaze in the face of unknown perturbations of the body, selective attention, and the complexity of stereo vision. In this article, we suggest control circuits to realize three of the most basic oculomotor behaviors and their integration: the vestibulo-ocular reflex and optokinetic response (VOR–OKR) for gaze stabilization, smooth pursuit for tracking moving objects, and saccades for overt visual attention. Each of these behaviors and the mechanism for their integration were derived with inspiration from computational theories as well as behavioral and physiological data in neuroscience. Our implementations on humanoid demonstrate good performance of oculomotor behaviors, which proves to be a viable strategy to explore novel control mechanisms for humanoid robotics. Conversely, insights gained from our models have been able to directly influence views and provide new directions for computational neuroscience research.

One of the key computations performed in human brain is multi-sensory cue integration, through which humans are able to estimate the current state of the world to discover relative reliabilities and relations between observed cues. Mammalian cortex consists of highly distributed and interconnected populations of neurons, each providing a specific type of information about the state of the world. Connections between areas seemingly realize functional relationships amongst them and computation occurs by each area trying to be consistent with the areas it is connected to. In this paper using line-attraction dynamics and divisive normalization, we present a computational framework which is able to learn arbitrary non-linear relations between multiple cues using a simple Hebbian Learning principle. After learning, the network dynamics converges to the stable state so to satisfy the relation between connected populations. This network can perform several principle computational tasks such as inference, de-noising and cue-integration. By applying a real world multi-sensory integrating scenario, we demonstrate that the network can encode relative reliabilities of cues in different areas of the state space, over distributed population vectors. This reliability based encoding biases the network’s dynamics in favor of more reliable cues and realizes a near optimal sensory integration mechanism. Additional important features of the network are its scalability to cases with higher order of modalities and its flexibility to learn smooth functions of relations which is necessary for a system to operate in a dynamic environment.

We present a novel system for vision based anomaly detection in real-time environments. Our system uses an event-based vision sensor consisting of asynchronously operating pixels that is inspired by the human retina. Each pixel reports events of illumination changes, are processed in a purely event-based tracker that pursues edges of events in the input stream. The tracker estimates are used to determine whether the input events originate from anomalous or regular data. We distinguish between the two cases with a Growing Neural Gas (GNG), which is modified to suite our event-based processing pipeline. While learning of the GNG is supervised and performed offline, the detection is carried out online. We evaluate our system by inspection of fast-spinning cog-wheels. Our system achieves faster than real-time speed on commodity hardware and generalizes well to other cases. The results of this paper can be applied both to technical implementations where high speed but little processing power is required, and for further investigations into event-based algorithms.

Human mind from system perspective deals with high dimensional complex world as an adaptive Multi-Input Multi-Output complex system. This view is theorized by reductionism theory in philosophy of mind, where the world is represented as logical combination of simpler sub-systems for human so that operate with less energy. On the other hand, Human usually uses linguistic rules to describe and manipulate his expert knowledge about the world; the way that is well modeled by Fuzzy Logic. But how such a symbolic form of knowledge can be encoded and stored in plausible neural circuitry? Based on mentioned postulates, we have proposed an adaptive Neuro-Fuzzy machine in order to model a rule-based MIMO system as logical combination of spatially distributed Single-Input Single-Output sub-systems. Each SISO systems as sensory and processing layer of the inference system, construct a single rule and learning process is handled by a Hebbian-like Spike-Time Dependent Plasticity. To shape a concrete knowledge about the whole system, extracted features of SISO neural systems (or equivalently the rules associated with SISO systems) are combined. To exhibit the system applicability, a single link cart-pole balancer as a sensory-motor learning task, has been simulated. The system is provided by reinforcement feedback from environment and is able to learn how to get expert and achieve a successful policy to perform motor control.

The SpiNNaker neural computing project has created a hardware architecture capable of scaling up to a system with more than a million embedded cores, in order to simulate more than one billion spiking neurons in biological real time. The heart of this system is the SpiNNaker chip, a multi-processor System-on-Chip with a high level of interconnectivity between its processing units. Here we present a Dynamically Extendable SpiNNaker Chip Computing Module that allows a SpiNNaker machine to be deployed on small mobile robots. A non-neural application, the simulation of the movement of a flock of birds, was developed to demonstrate the general purpose capabilities of this new platform. The developed SpiNNaker machine allows the simulation of up to one million spiking neurons in real time with a single SpiNNaker chip and is scalable up to 256 computing nodes in its current state.

When working with probabilistic graphical models we usually have two options to build the model: either using a Bayesian network (BN) or a Markov random field (MRF). However, there exist one more graphical representation which is able to unify the properties of BN and MRF that is called Factor Graph. This paper describes conceptual methods in working with factor graph especially with discrete random variables, how to learn its parameter and how to perform inference for making a reasoning task with it. Here we use population coding principles to discretize continues values of messages transmitted within the factor graph to update the network's internal belief. We provide several illustrative examples to highlight important aspects when developing a model for factor graphs.

In this paper we present an autonomous mobile robot setting that automatically explores and maps unknown indoor environments, exclusively with information from an embedded event-based dynamic vision sensor (eDVS) and a ring of bump switches on the robot. The eDVS provides a sparse pre-processed visual signature of the currently visible patch of ceiling, which is used for real-time simultaneous localization and mapping (SLAM). Signals from the robot's bump switches together with its current position estimate continuously improve the system's reasoning about traversable areas. A heuristic path planning method motivated by the A* search algorithm generates routes for continuous autonomous exploration. We demonstrate robust real-time operation and evaluate the performance of our system in various indoor environments.

The sensorimotor maps link the perceived states to actions, required to achieve the goals of a behaving agent. These mappings depend on the physics of the body of the agent, as well as the dynamics and geometry of the environment, in which the behavior unfolds. Autonomous acquisition and updating of the mappings is crucial for robust behavior in a changing real-world environment. Autonomy of many architectures, which implement the learning and adaptation of sensorimotor maps, is limited. Here, we present a neural-dynamic architecture that enables autonomous learning of the sensorimotor transformation, involved in looking behavior. The architecture is built using Dynamic Neural Fields and is implemented on a robotic agent that consists of an eDVS sensor on a pan-tilt unit.

We propose a novel method for vision based simultaneous localization and mapping (vSLAM) using a biologically inspired vision sensor that mimics the human retina. The sensor consists of a 128x128 array of asynchronously operating pixels, which independently emit events upon a temporal illumination change. Such a representation generates small amounts of data with high temporal precision; however, most classic computer vision algorithms need to be reworked as they require full RGB(-D) images at fixed frame rates. Our presented vSLAM algorithm operates on individual pixel events and generates high-quality 2D environmental maps with precise robot localizations. We evaluate our method with a state-of-the-art marker-based external tracking system and demonstrate real-time performance on standard computing hardware.