Search

The flexibility and dexterity of human limbs rely on the processing of a vast quantity of signals within the sensory-motor networks in the brain and spinal cord, distilled into stimuli that govern the commands and movements. Hence, the use of assistive devices, such as robotic limbs or exoskeletons, is critically dependent on the processing of a large number of heterogeneous signals to mimic natural movements. This article provides a panoramic overview of the three paradigms for the control of bionic limbs based on mechatronic technology. Two of them have already been established in the literature, while the third one, advocated by this article, is an emerging approach, enabled by the latest developments in connectivity and computation. In the first paradigm, the bionic limbs rely on conventional control and are directly reconnected to the human sensory-motor system, which requires a large signal processing bandwidth. The second paradigm is based on semiautonomous limbs, endowed with context-aware processing and certain decision capability. Following the advances in wireless connectivity and cloud/edge processing, this article introduces a third paradigm of connected limbs.

Electrotactile stimulation can be an attractive technology to restore tactile feedback in different application scenarios (e.g., virtual and augmented reality, tele-manipulation). This technology allows designing compact solutions with no mechanical elements that can integrate a high-density matrix of stimulation points. The present study introduced four novel multi-pad finger-electrode designs with different arrangements (two matrix and two circular) and shapes of active pads (producing sensation) and reference pads (ideally, no sensation produced below the pad). The electrodes were used to investigate the subjects’ ability to spatially discriminate active pads within phalanges individually (6–9 pads) as well as across the full finger (18–19 pads). The tests were conducted in 12 subjects and the results showed that all designs led to high success rates when applied to the fingertip (70–81%). When tested on the full finger, the matrix and circular designs were characterized with similar performance (54–57%), and when the phalanges were analyzed individually, the spatial discrimination was best at the fingertip. Additionally, new approaches for faster amplitude calibration were proposed and tested, demonstrating that calibration duration can be reduced by approximately 40% compared to the standard approach of calibrating single pads individually. Finally, discrimination tests of dynamic tactile patterns were conducted using circular and matrix designs on the fingertip and full finger, respectively. The tests showed that the different patterns generated by the two arrangements could be clearly discriminated, especially in the case of full-finger matrix-style patterns. The present study, therefore, provides several important insights that are relevant when delivering tactile feedback to the finger using an electrotactile interface.

Affordance segmentation is used to split object images into parts according to the possible interactions, usually to drive safe robotic grasping. Most approaches to affordance segmentation are computationally demanding; this hinders their integration into wearable robots, whose compact structure typically offers limited processing power. This article describes a design strategy for tiny, deep neural networks (DNNs) that can accomplish affordance segmentation and deploy effectively on microcontroller-like processing units. This is attained by specialized, hardware-aware neural architecture search (HW-NAS). The method was validated by assessing the performance of several tiny networks, at different levels of complexity, on three benchmark datasets. The outcome measure was the accuracy of the generated affordance maps and the associated spatial object descriptors (orientation, center of mass, and size). The experimental results confirmed that the proposed method compared satisfactorily with the state-of-the-art approaches, yet allowing a considerable reduction in both network complexity and inference time. The proposed networks can, therefore, support the development of a teleceptive sensing system to improve the semiautomatic control of wearable robots for assisting grasping.

When using EMG biofeedback to control the grasping force of a myoelectric prosthesis, subjects need to activate their muscles and maintain the myoelectric signal within an appropriate interval. However, their performance decreases for higher forces, because the myoelectric signal is more variable for stronger contractions. Therefore, the present study proposes to implement EMG biofeedback using nonlinear mapping, in which EMG intervals of increasing size are mapped to equal-sized intervals of the prosthesis velocity. To validate this approach, 20 non-disabled subjects performed force-matching tasks using Michelangelo prosthesis with and without EMG biofeedback with linear and nonlinear mapping. Additionally, four transradial amputees performed a functional task in the same feedback and mapping conditions. The success rate in producing desired force was significantly higher with feedback (65.4±15.9%) compared to no feedback (46.2±14.9%) as well as when using nonlinear (62.4±16.8%) versus linear mapping (49.2±17.2%). Overall, in non-disabled subjects, the highest success rate was obtained when EMG biofeedback was combined with nonlinear mapping (72%), and the opposite for linear mapping with no feedback (39.6%). The same trend was registered also in four amputee subjects. Therefore, EMG biofeedback improved prosthesis force control, especially when combined with nonlinear mapping, which showed to be an effective approach to counteract increasing variability of myoelectric signal for stronger contractions.

The stringent timing and reliability requirements in mission-critical applications require a detailed statistical characterization of end-to-end latency. Teleoperation is a representative use case, in which a human operator (HO) remotely controls a robot by exchanging command and feedback signals. We present a framework to analyze the latency of a closed-loop teleoperation system consisting of three entities: an HO, a robot located in remote environment, and a Base Station (BS) with Mobile edge Computing (MEC) capabilities. A model of each component is used to analyze the closed-loop latency and optimize the compression strategy. The closed-form expression of the distribution of the closed-loop latency is difficult to estimate, such that suitable upper and lower bounds are obtained. We formulate a non-convex optimization problem to minimize the closed-loop latency. Using the obtained upper and lower bound on the closed-loop latency, a computationally efficient procedure to optimize the closed-loop latency is presented. The simulation results reveal that compression of sensing data is not always beneficial, while system design based on average performance leads to under-provisioning and may cause performance degradation. The applicability of the proposed analysis is much wider than teleoperation, including a large class of systems whose latency budget consists of many components.

Humans systematically adapt their strategies for closed-loop control based on visual feedback according to the dynamics of the system. Tactile feedback is a key element in many human–machine interfaces, but it is not known if and how well human control adapts to changes in system dynamics when information about the system state is provided using this type of feedback. In this study, 11 participants tracked a pseudorandom trajectory with a virtual, position- or velocity-controlled plant using a joystick. Visual or electrotactile feedback provided the instantaneous error between the target and generated trajectory. Frequency-domain system identification indicated that human control adapted in similar ways to the different control modes (i.e., position/velocity control) for both feedback modalities. For the plant dynamics modeled as gain and integrator, the human controller behaved as a low-pass filter and gain, respectively (under the assumption of quasi-linear behavior). However, while tracking quality was largely similar for both control modes with visual feedback, velocity control enabled substantially worse control with electrotactile feedback compared to position control. Furthermore, for both control modes, the crossover frequency of open-loop transfer functions was lower for electrotactile feedback (0.9 and 1.1 rad/s) than for visual feedback (1.5 and 1.7 rad/s) indicating limited control bandwidth. To summarize, closed-loop control based on electrotactile feedback enables natural adaptations in human control strategy, which is encouraging for tactile feedback-controlled human–machine interfaces, but the lower control bandwidth and lower tracking quality with velocity control may impose functional limitations.

Just-noticeable difference (JND), indicating the ability to accurately identify small differences in stimulation parameters, can be used to choose more sensitive stimulation methods as well as to calibrate tactile feedback in closed-loop human-machine interfacing. The JND is typically estimated using a forced-choice-discrimination task, in which two stimuli with different intensities are delivered separated by a brief pause. In the applications of tactile feedback, however, the stimulation parameters are typically modulated continuously. It is unclear if the discriminability of stimuli separated in time characterizes the ability to distinguish continuous changes in stimulation intensity. The present study compared the JND when pairs of frequency-modulated electrotactile stimuli were separated in time and presented continuously at two different baseline frequencies (20 and 60 Hz). The results showed that the JND was significantly smaller with time-separation between stimuli, but that the JND obtained with different types of transitions were in most cases linearly associated. In conclusion, the discriminability of time-separated stimuli is systematically better compared to that of the stimuli presented continuously. This can have an impact when calibrating the tactile feedback where the conventional method of the JND assessment might lead to an overly optimistic estimate of detectable changes.

The performance of a classifier in a brain–computer interface (BCI) system is highly dependent on the quality and quantity of training data. Typically, the training data are collected in a laboratory where the users perform tasks in a controlled environment. However, users’ attention may be diverted in real-life BCI applications and this may decrease the performance of the classifier. To improve the robustness of the classifier, additional data can be acquired in such conditions, but it is not practical to record electroencephalogram (EEG) data over several long calibration sessions. A potentially time- and cost-efficient solution is artificial data generation. Hence, in this study, we proposed a framework based on the deep convolutional generative adversarial networks (DCGANs) for generating artificial EEG to augment the training set in order to improve the performance of a BCI classifier. To make a comparative investigation, we designed a motor task experiment with diverted and focused attention conditions. We used an end-to-end deep convolutional neural network for classification between movement intention and rest using the data from 14 subjects. The results from the leave-one subject-out (LOO) classification yielded baseline accuracies of 73.04% for diverted attention and 80.09% for focused attention without data augmentation. Using the proposed DCGANs-based framework for augmentation, the results yielded a significant improvement of 7.32% for diverted attention ( $p < 0.01$ ) and 5.45% for focused attention ( $p < 0.01$ ). In addition, we implemented the method on the data set IVa from BCI competition III to distinguish different motor imagery tasks. The proposed method increased the accuracy by 3.57% ( $p < 0.02$ ). This study shows that using GANs for EEG augmentation can significantly improve BCI performance, especially in real-life applications, whereby users’ attention may be diverted.

Tactile feedback is pivotal for grasping and manipulation in humans. Providing functionally effective sensory feedback to prostheses users is an open challenge. Past paradigms were mostly based on vibro- or electrotactile stimulations. However, the tactile sensitivity on the targeted body parts (usually the forearm) is greatly less than that of the hand/fingertips, restricting the amount of information that can be provided through this channel. Visual feedback is the most investigated technique in motor learning studies, where it showed positive effects in learning both simple and complex tasks; however, it was not exploited in prosthetics due to technological limitations. Here, we investigated if visual information provided in the form of augmented reality (AR) feedback can be integrated by able-bodied participants in their sensorimotor control of a pick-and-lift task while controlling a robotic hand. For this purpose, we provided visual continuous feedback related to grip force and hand closure to the participants. Each variable was mapped to the length of one of the two ellipse axes visualized on the screen of wearable single-eye display AR glasses. We observed changes in behavior when subtle (i.e., not announced to the participants) manipulation of the AR feedback was introduced, which indicated that the participants integrated the artificial feedback within the sensorimotor control of the task. These results demonstrate that it is possible to deliver effective information through AR feedback in a compact and wearable fashion. This feedback modality may be exploited for delivering sensory feedback to amputees in a clinical scenario.