Search

This study presents a simple Human Machine Interface (HMI) for gesture recognition purpose, based on a wireless piezoresistive armband. The armband embeds three sensors based on Force Sensitive Resistors (FSRs) applied on specific forearm muscles, which provide signals comparable to the electromyography linear envelope. The system aims to recognize in real-time some hand gestures, opportunely processing the force signals. The HMI control system is based on Arduino platform and implements a Linear Discriminant Analysis (LDA) classifier to perform real-time gesture recognition. The HMI, by means of a Bluetooth module can wireless connect to a computer and provide commands to custom graphical interfaces or other applications as videogames. Four healthy volunteers were involved in testing the device. A double threshold segmentation technique, applied to the recorded FSR signals, allowed to detect and extract features regarding the whole-time evolution of a performed gesture, explicitly considering the transition between gestures. LDA classification performances were assessed by applying the 10-fold cross validation. The mean accuracy across all participants resulted 93.48%. The results suggest that the HMI device allows easy user's interaction with a computer in real-time. In prospective, this device could find practical applications in neuromotor rehabilitation (e.g. exergaming), and in controlling prosthetic hands.

Optical see-through automotive head-up displays (HUDs) are a form of augmented reality (AR) that is quickly gaining penetration into the consumer market. Despite increasing adoption, demand, and competition among manufacturers to deliver higher quality HUDs with increased fields of view, little work has been done to understand how best to design and assess AR HUD user interfaces, and how to quantify their effects on driver behavior, performance, and ultimately safety. This paper reports on a novel, low-cost, immersive driving simulator created using a myriad of custom hardware and software technologies specifically to examine basic and applied research questions related to AR HUDs usage when driving. We describe our experiences developing simulator hardware and software and detail a user study that examines driver performance, visual attention, and preferences using two AR navigation interfaces. Results suggest that conformal AR graphics may not be inherently better than other HUD interfaces. We include lessons learned from our simulator development experiences, results of the user study and conclude with limitations and future work. National Science FoundationNational Science Foundation (NSF) [1816721] This material is based upon work supported by the National Science Foundation under Grant No. 1816721.

The goal of this work is to present a novel continuous finger gesture recognition system based on flex sensors. The system is able to carry out accurate recognition of a sequence of gestures. Wireless smart gloves equipped with flex sensors were implemented for the collection of the training and testing sets. Given the sensory data acquired from the smart gloves, the gated recurrent unit (GRU) algorithm was then adopted for gesture spotting. During the training process for the GRU, the movements associated with different fingers and the transitions between two successive gestures were taken into consideration. On the basis of the gesture spotting results, the maximum a posteriori (MAP) estimation was carried out for the final gesture classification. Because of the effectiveness of the proposed spotting scheme, accurate gesture recognition was achieved even for complicated transitions between successive gestures. From the experimental results, it can be observed that the proposed system is an effective alternative for robust recognition of a sequence of finger gestures.

During the last years, the mobility of people with upper limb disabilities and constrained on power wheelchairs is empowered by robotic arms. Nowadays, even though modern manipulators offer a high number of functionalities, some users cannot exploit all those potentialities due to their reduced manual skills, even if capable of driving the wheelchair by means of proper Human&ndash, Machine Interface (HMI). Owing to that, this work proposes a low-cost manipulator realizing only simple tasks and controllable by three different graphical HMI. The latter are empowered using a You Only Look Once (YOLO) v2 Convolutional Neural Network that analyzes the video stream generated by a camera placed on the robotic arm end-effector and recognizes the objects with which the user can interact. Such objects are shown to the user in the HMI surrounded by a bounding box. When the user selects one of the recognized objects, the target position information is exploited by an automatic close-feedback algorithm which leads the manipulator to automatically perform the desired task. A test procedure showed that the accuracy in reaching the desired target is 78%. The produced HMIs were appreciated by different user categories, obtaining a mean score of 8.13/10.

Devices that track the human body movement are heavily used in numerous and various fields like medicine, automation and entertainment. The work proposed is focused on the design of a modular device able track the flection of human hand phalanxes. The overall system composed by two parts: a computer program interface and a modular wearable system applied to the finger whose motion is to be monitored. The wearable device is equipped with an Inertial Motion Unit (IMU) with the purpose to detect the first phalanx orientation and a stretch sensor applied between the first and the second phalanx to recognize the flection angle. The configuration is completed with a microcontroller unit (ATmega328P) and a Bluetooth Low Power Module (RN4871) to ensure a reliable and easy to implement communication channel. We conduct two main set of tests to verify the global functionalities. In the first set the device is used to track the full flexion of a single finger while in the second we test the device capability to recognize different grabbed objects starting from the data retrieved from two fingers. The preliminary results open the possibility of a future development focused on a modular device composed by five elements, one for each hand finger and able to detect complex gesture like pinch, spread or tap.

The increment of the prevalence of neurological diseases due to the trend in population aging demands for new strategies in disease management. In Parkinson&rsquo, s disease (PD), these strategies should aim at improving diagnosis accuracy and frequency of the clinical follow-up by means of decentralized cost-effective solutions. In this context, a system suitable for the remote monitoring of PD subjects is presented. It consists of the integration of two approaches investigated in our previous works, each one appropriate for the movement analysis of specific parts of the body: low-cost optical devices for the upper limbs and wearable sensors for the lower ones. The system performs the automated assessments of six motor tasks of the unified Parkinson&rsquo, s disease rating scale, and it is equipped with a gesture-based human machine interface designed to facilitate the user interaction and the system management. The usability of the system has been evaluated by means of standard questionnaires, and the accuracy of the automated assessment has been verified experimentally. The results demonstrate that the proposed solution represents a substantial improvement in PD assessment respect to the former two approaches treated separately, and a new example of an accurate, feasible and cost-effective mean for the decentralized management of PD.

Body motion tracking technologies are widespread in military, medical and sport fields. The work proposes a modular wireless wearable system able to detect hand fingers motion. Such system is composed by a readout unit that analyses the data and by a wearable measuring device directly applied on the tracked finger. This device is equipped with an Inertial Motion Unit (IMU) used to track the first phalanx motion and orientation and with a stretch sensor to monitor the bending angle between the first and second phalanxes. We carried out an experimental study, which is divided in two main parts. In the first one, the transducer performance was evaluated, whereas, in the second part, we tested the capability of the overall system to recognize simple finger movements and different objects grabbed. The preliminary results pave the possibility of developing a modular device, one for each hand finger, able to recognize the grabbed object shape or detect complex gestures.

Injuries, accidents, strokes, and other diseases can significantly degrade the capabilities to perform even the most simple activities in daily life. While assistive technology becomes more and more available to the people affected, there is still a big need for user interfaces suitable for people without functional hand movement. A large share of these cases involves neuromuscular diseases, which lead to severely reduced muscle function. However, even though affected people are no longer able to functionally move their limbs, residual muscle function can still be existent. Previous work has shown that this residual muscular activity can suffice, to create an EMG-based user interface, and e.g. allow for control of assistive devices. In this paper, we enhance this user interface with additional EMG-features and an improved training paradigm in order to increase information extraction from recordings of strongly atrophic muscles. The interface was tested and validated by subjects with severe spinal muscular atrophy. Results show that the used methods improve the decoding and thereby allow for a considerable increase in performance when controlling a robotic manipulator in a 3D reaching task.

The recent increase in the spreading of Virtual Reality allows a large number of consumers to experience this technology in several fields, from gaming to learning. In this paper we propose a framework that allows the user to freely move and interact in a virtual museum implemented with the HTC Vive. The goals of this work are: to investigate the possibility of using this approach for providing the user with a personalized and educative exploration of an artistic content, and to measure the immersivity and usability of the proposed system by means of a subjective experiment.

Physiological devices (PDs) like upright bicycles, steppers and treadmills act as Human Machine Interface (HMI) during rehabilitation. The main issue on how to utilize biosignals such as Electromyography (EMG), Electrocardiography (ECG) and Electroencephalography (EEG) as an inputs for HMI to control the PDs. Biosignals are stochastic and complex as they influenced by anatomical and physiological properties of muscles. To maximize the training time during exercise, the features of the biosignals (e.g., fatigue, contraction or relaxation) should be extracted to maintain the system as a reliable working condition. For that purpose, it is feasible to try out a probabilistic distribution as a feature to illustrate the pattern of muscle activation. Before the estimation of parameter distribution is conducted, we need to verify the types of distribution that fit the raw biosignals. In this study, EMG and EEG signals will be considered to find the most suitable distribution for the signals. The selected model is chosen based on a minimum error produced by two Goodness-of-Fit (GOF) tests namely Kolmogorov-Smirnov statistic, D and Anderson Darling statistic, A 2. As a result, a Generalized Extreme Value (GEV) distribution is found the most appropriate distribution compared to Generalized Pareto (GP) and Exponential (EXP) distributions for describing the EMG and EEG signals.

The goal of this study is to improve future physical human-robot interaction and rehabilitation systems. Experiments were conducted to collect dominant hand grip pressure and joint-angle data during activities of daily life. Representative actions chosen as part of this study were: pushing a weighted cylinder along a flat surface, pulling a weighted cylinder across a flat surface, and lifting a weighted cylinder from a flat surface to shoulder height. Three separate weighted cylinders were used, 3lbs., 5lbs., and 10lbs., and the representative motions were repeated five times for each cylinder. A Tekscan Grip VersaTek Pressure Measurement System and Motion Analysis Cortex System were utilized to collect data. Each subject was outfitted with 18 separate sensorized piezo-resistive tiles placed on their dominant hand and 33 reflective markers at representative locations on their body. The motion of each cylinder was tracked via the placement of seven retro-reflective markers on the cylinder's surface. Analysis of cohort data from five male and five female volunteers, aged between 23 and 51 years, is presented. A Moving Average Filter was implemented to automatically determine contact between the subject's hand and the weighted cylinder. Once contact was determined during an action cycle, maximum detected pressure from each of 18 sensing areas was found. Results report wrist angle during the action- cycle as well as maximum applied pressure during each action across the cohort. Average wrist angle per action-cycle, by action, is also reported for the cohort. These data, along with results from a previous study, will be used improve and verify human intent models for use in future pHRI and rehabilitation systems.

Road transport plays a significant role in various industries and mobility services around the globe and has a vital impact on our daily lives. However it also has serious impacts on both public health and the environment. In-vehicle feedback systems are a relatively new approach to encouraging driver behavior change for improving fuel efficiency and safety in automotive environments. While many studies claim that the adoption of eco-driving practices, such as eco-driving training programs and in-vehicle feedback to drivers, has the potential to improve fuel efficiency, limited research has integrated safety and eco-driving. Therefore, it is crucial to understand the human factors related theories and practices which will inform the design of an in-vehicle Human Machine Interface (HMI) that could provide real-time driver feedback and consequently improve both fuel efficiency and safety. This paper provides a comprehensive review of the current state of published literature on in-vehicle systems to identify and evaluate the impact of eco-driving and safety feedback systems. This paper also discusses how these factors may conflict with one another and have a negative effect on road safety, while also exploring possible eco-driving practices that could encourage more sustainable, environmentally-conscious and safe driving behavior. The review revealed a lack of comprehensive theoretical research integrating eco-driving and safe driving, and no current available HMI covering both aspects simultaneously. Furthermore, the review identified that some eco-driving in-vehicle systems may enhance fuel efficiency without compromising safety. The review has identified a range of concepts which can be developed to influence driver acceptance of safety and eco-driving systems within the area of HMI. This can promote new research aimed at enhancing our understanding of the relationship between eco-driving and safety from the human factors viewpoint. This provides a foundation for developing innovative, persuasive and acceptable in-vehicle HMI systems to improve fuel efficiency and road safety.

Motions of the fingers are complex since hand grasping and manipulation are conducted by spatial and temporal coordination of forearm muscles and tendons. The dominant methods based on surface electromyography (sEMG) could not offer satisfactory solutions for finger motion classification due to its inherent nature of measuring the electrical activity of motor units at the skin's surface. In order to recognize morphological changes of forearm muscles for accurate hand motion prediction, ultrasound imaging is employed to investigate the feasibility of detecting mechanical deformation of deep muscle compartments in potential clinical applications. In this study, finger motion classification has been represented as subproblems: recognizing the discrete finger motions and predicting the continuous finger angles. Predefined 14 finger motions are presented in both sEMG signals and ultrasound images and captured simultaneously. Linear discriminant analysis classifier shows the ultrasound has better average accuracy (95.88%) than the sEMG (90.14%). On the other hand, the study of predicting the metacarpophalangeal (MCP) joint angle of each finger in nonperiod movements also confirms that classification method based on ultrasound achieves better results (average correlation 0.89 $\pm$ 0.07 and NRMSE 0.15 $\pm$ 0.05) than sEMG (0.81 $\pm$ 0.09 and 0.19 $\pm$ 0.05). The research outcomes evidently demonstrate that the ultrasound can be a feasible solution for muscle-driven machine interface, such as accurate finger motion control of prostheses and wearable robotic devices.

In the last decades, smart power wheelchairs have being used by people with motor skill impairment in order to improve their autonomy, independence and quality of life. The most recent power wheelchairs feature many technological devices, such as laser scanners to provide automatic obstacle detection or robotic arms to perform simple operations like pick and place. However, if a motor skill impaired user was able to control a very complex robotic arm, paradoxically he would not need it. For that reason, in this paper we present an autonomous control system based on Computer Vision algorithms which allows the user to interact with buttons or elevator panels via a robotic arm in a simple and easy way. Scale-Invariant Feature Transform (SIFT) algorithm has been used to detect and track buttons. Objects detected by SIFT are mapped in a tridimensional reference system collected with Parallel and Tracking Mapping (PTAM) algorithm. Real word coordinates are obtained using a Maximum-Likelihood estimator, fusing the PTAM coordinates with distance information provided by a proximity sensor. The visual servoing algorithm has been developed in Robotic Operative System (ROS) Environment, in which the previous algorithms are implemented as different nodes. Performances have been analyzed in a test scenario, obtaining good results on the real position of the selected objects.

In the frame of assistive robotics, multi-finger prosthetic hand/wrists have recently appeared, offering an increasing level of dexterity; however, in practice their control is limited to a few hand grips and still unreliable, with the effect that pattern recognition has not yet appeared in the clinical environment. According to the scientific community, one of the keys to improve the situation is multi-modal sensing, i.e., using diverse sensor modalities to interpret the subject's intent and improve the reliability and safety of the control system in daily life activities. In this work, we first describe and test a novel wireless, wearable force- and electromyography device; through an experiment conducted on ten intact subjects, we then compare the obtained signals both qualitatively and quantitatively, highlighting their advantages and disadvantages. Our results indicate that force-myography yields signals which are more stable across time during whenever a pattern is held, than those obtained by electromyography. We speculate that fusion of the two modalities might be advantageous to improve the reliability of myocontrol in the near future.

Wheelchair Mounted Robotic Arms (WMRA) can be used by people with severe motor skill impairment, such as SMA (Spinal Muscular Atrophy), Cerebral Palsy etc…, in order to achieve daily life tasks. Many of those systems have been presented in literature and are available on the market but they are really expensive and bulky.

One of the crucial problems found in the scientific community of assistive / rehabilitation robotics nowadays is that of automatically detecting what a disabled subject (for instance, a hand amputee) wants to do, exactly when she wants to do it and strictly for the time she wants to do it. This problem, commonly called intent detection, has traditionally been tackled using surface electromyography, a technique which suffers from a number of drawbacks, including the changes in the signal induced by sweat and muscle fatigue. With the advent of realistic, physically plausible augmented- and virtual-reality environments for rehabilitation, this approach does not suffice anymore. In this paper we explore a novel method to solve the problem, that we call Optical Myography (OMG). The idea is to visually inspect the human forearm (or stump) to reconstruct what fingers are moving and to what extent. In a psychophysical experiment involving ten intact subjects, we used visual fiducial markers (AprilTags) and a standard web-camera to visualize the deformations of the surface of the forearm, which then were mapped to the intended finger motions. As ground truth, a visual stimulus was used, avoiding the need for finger sensors (force/position sensors, datagloves, etc.). Two machine-learning approaches, a linear and a non-linear one, were comparatively tested in settings of increasing realism. The results indicate an average error in the range of 0.05 to 0.22 (root mean square error normalized over the signal range), in line with similar results obtained with more mature techniques such as electromyography. If further successfully tested in the large, this approach could lead to vision-based intent detection of amputees, with the main application of letting such disabled persons dexterously and reliably interact in an augmented- / virtual-reality setup.

This doctoral thesis makes an in-depth analysis of HRV (Heart Rate Variability) as calculated from an ECG (Electro Cardio Gram) and how it varies related to the attention span shown by individuals when driving land vehicles. The aim is to use this knowledge in applications where this attention span needs to be estimated, and to focus primarily on measuring the influence that the new HMIs (Human Machine Interfaces) being tried out in the latest generation of vehicles may have on road safety. More specifically, the thesis concentrates on the identification of those parameters which best relate to variability due to personal characteristics such as age, temporary features such as circadian cycles, or the surrounding environment as in road category, in order to integrate all of these features into a complete system. It is this integrating feature which is most lacking in the literature, indeed it is almost non-existent, so that the present work seeks to achieve the most universal classification criteria. This is essential for its use in applications such as vehicle driving, which require a constant state of alert. To this end, an application has been developed which allows patterns to be generated according to a system of four standard variables derived from the HRV, allowing different ways of interacting with on-road vehicles to be compared and assessed. Finally, a further point of note is that the analysis contained in this thesis may be extended to other fields (e.g. risky activities or environments) where it is useful or indeed necessary to take a relative measurement of attention span rather than follow a simple patterns, as understood by the Kahneman model.

In the last few years power wheelchairs have been becoming the only device able to provide autonomy and independence to people with motor skill impairments. In particular, many power wheelchairs feature robotic arms for gesture emulation, like the interaction with objects. However, complex robotic arms often require a joystic to be controlled; this feature make the arm hard to be controlled by impaired users. Paradoxically, if the user were able to proficiently control such devices, he would not need them. For that reason, this paper presents a highly autonomous robotic arm, designed in order to minimize the effort necessary for the control of the arm. In order to do that, the arm feature an easy to use human - machine interface and is controlled by Computer Vison algorithm, implementing a Position Based Visual Servoing (PBVS) control. It was realized by extracting features by the camera and fusing them with the distance from the target, obtained by a proximity sensor. The Parallel Tracking and Mapping (PTAM) algorithm was used to find the 3D position of the task object in the camera reference system. The visual servoing algorithm was implemented in an embedded platform, in real time. Each part of the control loop was developed in Robotic Operative System (ROS) Environment, which allows to implement the previous algorithms as different nodes. Theoretical analysis, simulations and in system measurements proved the effectiveness of the proposed solution.

The paper describes a system able to recognize the users identity according how she/he looks at the monitor while using a given interface. The system does not need invasive measurements that could limit the naturalness of her/his actions. The proposed approach clusters the sequences of observed points on the screen and characterizes the user identity according the relevant detected patterns. Moreover, the system is able to identify patterns in order to have a more accurate recognition and to create prototypes of natural facial dynamics in user expressions. The possibility to characterize people through facial movements introduces a new perspective on human-machine interaction. For example, a user can obtain different contents according her/his mood or a software interface can modify itself to keep a higher attention from a bored user. The success rate of the classification using only 7 parameters is around 68%. The approach is based on k-means that is tuned to maximize an index involving the number of true-positive detections and conditional probabilities. A different evaluation of this parameter allows to focus on the identification of a single user or to spot a general movement for a wide range of people The experiments show that the performance can reach the 90% of correct recognition.

The article of record as published may be found at http://dx.doi.org/10.1016/j.neucom.2011.09.018 This study aimed to improve the accuracy and robustness of a real-time assistive human machine interface system by classifying between the controlled movements related tongue-movement ear pressure (TMEP) signals and the interfering signals. The controlled movement TMEP signals were collected during left, right, up, down, flicking and pushing tongue motions. The TMEP signals were processed and classified using detection, segmentation, feature extraction and classification. The segmented signals were decomposed into the time-scale domain using a wavelet packet transform. The variance of the wavelet packet coefficients and its ratio between low-to-high scales were defined as features and the intended tongue movement commands and interfering signals were classified using both a Bayesian and support vector machine (SVM) classifiers for comparison. The average classification accuracy for discriminating between the controlled movements and the interfering signals achieved 97.8% (Bayesian) and 98.5% (SVM). The classifiers were robust remaining at a similar performance level when generalised interferences from all subjects were used. It was shown that the Bayesian classifier performed better than the SVM in a real-time environment. The approach of combining the Bayesian classifier and the wavelet packet transform provides a robust and efficient method for a real-time assistive human machine interface based on tongue-movement ear pressure signals. UK Engineering and Physical Sciences Research Council ISVR Rayleigh Scholarship, University of Southampton, UK (EPSRC; Grant number EP/F01869X/1)

In the Turing test, a computer model is deemed to "think intelligently" if it can generate answers that are not distinguishable from those of a human. However, this test is limited to the linguistic aspects of machine intelligence. A salient function of the brain is the control of movement, and the movement of the human hand is a sophisticated demonstration of this function. Therefore, we propose a Turing-like handshake test, for machine motor intelligence. We administer the test through a telerobotic system in which the interrogator is engaged in a task of holding a robotic stylus and interacting with another party (human or artificial). Instead of asking the interrogator whether the other party is a person or a computer program, we employ a two-alternative forced choice method and ask which of two systems is more human-like. We extract a quantitative grade for each model according to its resemblance to the human handshake motion and name it "Model Human-Likeness Grade" (MHLG). We present three methods to estimate the MHLG. (i) By calculating the proportion of subjects' answers that the model is more human-like than the human; (ii) By comparing two weighted sums of human and model handshakes we fit a psychometric curve and extract the point of subjective equality (PSE); (iii) By comparing a given model with a weighted sum of human and random signal, we fit a psychometric curve to the answers of the interrogator and extract the PSE for the weight of the human in the weighted sum. Altogether, we provide a protocol to test computational models of the human handshake. We believe that building a model is a necessary step in understanding any phenomenon and, in this case, in understanding the neural mechanisms responsible for the generation of the human handshake.

A CVSM (Complex-Valued Spiking Machine) is a tool intended for the processing of spatio-temporal patterns. This machine is built by assembling spiking neural networks named STANNs (Spatio-Temporal Artificial Neural Networks). The potentialities of the latter have been studied for several years in the field of human-computer interactions (handwritten character recognition, lip-reading and speech recognition). Whereas this field makes the object of a renewed interest with the development of mobile telephony and that of the personal digital assistants, the purpose of the experiment described in this paper was to evaluate the CVSM on an industrial application (it had been studied until then only in simulation). This application, designed to carry out the capture of electronic forms with a pen oriented interface, was done by students of Supelec for the account of France Telecom R&D. In addition to the results in handwritten character recognition, a detailed attention was granted during this study to the facility to integrate this technique in the traditional steps of a software development cycle.

This paper presents a new technique of data coding and an associated set of homogenous processing tools for the development of Human Computer Interactions (HCI). The proposed technique facilitates the fusion of different sensorial modalities and simplifies the implementations. The coding takes into account the spatio-temporal nature of the signals to be processed in the framework of a sparse representation of data. Neural networks adapted to such a representation of data are proposed to perform the recognition tasks. Their development is illustrated by two examples: one of on-line handwritten character recognition; and the other of visual speech recognition.

In this paper we focus on manual-to-automation and automation-to-manual transition scenarios in highway driving and the design of the instrument cluster (IC) in particular. Two alternative IC concepts were compared in order to find whether there are any differences in how easy they are to interpret and act according to. One IC contained two automation modes (manual and highly automated) while the other contained three modes (manual, Adaptive Cruise Control, highly automated). It was also hypothesized that traffic density may have an effect on how well the driver manages the transition scenarios. A simulator study was conducted in which 23 participants were exposed to the different IC designs (two vs. three modes) as well as the two degrees of traffic density (high, low). The main scenarios consisted of prompts for the driver when a higher degree of automation was possible to initiate, and different system-initiated takeover requests due to system limitations. The system limitations consisted in either just lateral automation being removed or both lateral and longitudinal automation being removed. Both subjective (e.g. general usefulness and intention to use) and objective measures (e.g. time to initiate a higher mode of automation) were used. The results showed that the two-mode IC resulted in faster takeover responses from the participants after full automation removal compared to when only lateral support was removed in the three-mode IC. The two-mode IC in some cases also resulted in quicker automation activation after an automation-available prompt.