Search

This paper showcases a developed end-user framework for a human-robot collaborative system for common tasks, such as pick and place. The system is designed for semi-automated pick and place tasks as well as manual operation making it flexible for multiple use-case scenarios. The goal of the system is to make the robot multi-functional, easy to use with a graphical user interface and to perform common tasks with the help of a human teammate. Integration with object recognition neural network (YoloV3) and an RGB-Depth camera helps automate pick and place tasks with a wide variety of objects.

Nurses have to carry out a myriad of tasks, which is one of the main reasons for burnout. In this paper, a robotic Multitasking Intelligent Nurse Aid (MINA) is proposed to assist nurses with everyday tasks and tackle nurse burnout. MINA uses Simultaneous Localization And Mapping (SLAM) to map shelves in a storage room and pointcloud using an RGB-Depth camera. The barcode of the item is first detected using zbar library, and then the pointcloud is generated in ROS and saved accordingly. Preliminary experiments with respect to the creation of a pointcloud map with barcode detection are presented. The preliminary results reveal that the system shows a 100% barcode detection rate, and with each barcode detection a corresponding pointcloud map is generated and saved automatically without any intervention.

In recent years, with the current advancements in Robotics and Artificial Intelligence (AI), robots have the potential to support the field of healthcare. Robotic systems are often introduced in the care of the elderly, children, and persons with disabilities, in hospitals, in rehabilitation and walking assistance, and other healthcare situations. In this survey paper, the recent advances in robotic technology applied in the healthcare domain are discussed. The paper provides detailed information about state-of-the-art research in care, hospital, assistive, rehabilitation, and walking assisting robots. The paper also discusses the open challenges healthcare robots face to be integrated into our society.

The purpose of this work is to detect and localize objects in a laboratory setting. For our experiments, we use fixed objects as reference points, based on which the robot achieves localization. We investigate the performance of the system on two classification categories in which labels were automatically acquired (humans, chairs).

Human-Robot Interaction often involves a robot assisting or providing feedback to a human partner's performance or cooperating to complete a task. In such an interaction scenario, the robotic system requires to perceive the human teammate's cognitive state that might affect task performance. In this pilot study, the focus is on developing a framework that assesses the human's cognitive performance for human-robot synergetic task, such as an assembly task. Specifically, we explore the correlation between a person's quality of sleep and performance metric through a standard task for cognitive assessment, the N-back task. To validate our hypothesis, we conducted a study with 25 participants, and our results indicate that there is a moderate correlation between some stages of sleep cycle and performance. Additionally, we present a possible Human-Robot Interaction setup that could benefit from our results.

Fatigue is a pervasive symptom following brain injury or disease. It has been known to impact recovery and hence is an important factor in rehabilitation. Robotic rehabilitation may be one way to reduce fatigue because of the robot's capability to adapt to user's performance. This paper explores an adaptive rehabilitation system to provide personalized upper limb rehabilitation. The system collects EMG data from the major muscles responsible for movement and adapts the forces used for rehabilitation (assistive and resistive) in real time based on muscle fatigue. Experimental results and the user survey outcomes show that the system was able to detect the onset of fatigue within ± 10 seconds error margin. Overall, it was found that the subjects experienced lower fatigue and had a higher probability of compliance and engagement with the proposed robotic rehabilitation system.

It is generally accepted in the field of human-robot interaction (HRI) that the role of a human teacher is critical for robot environmental adaptation. As such, the ability of a robot to adapt in a novel environment is based on the capabilities of the human teacher. In this paper, we propose a metric for robots to evaluate the quality of a human teacher. We illustrate a scenario where the Sawyer Robotic Arm must learn to grasp a cup according to the demonstration ofa good and a bad human teacher.

Objective: In this paper, we present a multimodal robot-based framework to investigate how physical and mental fatigue affect task performance, and how it relates to subjective self-reports.Methodology: In this pilot study, seven healthy participants underwent the robot-based assessment. In each session, the participants performed a series of reaching tasks, including a task with cognitive demands, by moving the end effector of the Barrett WAM arm in the direction of the virtual targets displayed on a computer screen. Multimodal data, including EEG, EMG and user performance data like reaction time, number of targets, trajectory of the robot path, were recorded for further analysis.Results: Based on the analysis of subjective user self-report and objective task performance metrics, we observe that the user's perceived level of task difficulty increased over time while objective task performance also improved over time. We speculate that this might be due to the effect of fatigue on the user's perception of task difficulty.Conclusion: Further studies are required, with a more diverse population, to understand the impact of fatigue on the user's cognitive and physical ability. We must also evaluate how contextual parameters may affect task performance and fatigue.

A considerable amount of research in the field of human-robot interaction has shown that a human teacher can be an integral component during the learning process of a robot. In this paper, we propose a learning framework that is based on learning from demonstration at a trajectory level. Specifically, we illustrate a scenario where the Sawyer Robotic Arm must learn to pick and place a specific object according to the demonstration of a human teacher. The purpose of the experiment is to facilitate the effectiveness of the proposed method.

Considerable amount of research is being carried out in robot based rehabilitation techniques. One of the main focus when building a smart rehabilitation system is its ability to adapt based on the user experience. In this poster we attempt to build such a smart rehabilitation system that recognizes the strain/ negative emotions from the participants facial expression and adjusts its force exerted. The accuracy of the system to recognize the facial expression is assessed and the accuracy of the system as a whole is estimated through user surveys.

In recent years robots have been replacing human workers in many industries. As robot manufacturing becomes more optimized and cost effective, the use of robots is going to evolve in various fields. Thus, it is essential to prepare workers to operate robots and work collaboratively with them side by side. In this paper, we are proposing a human-robot collaborative task to assess human performance in experimental, industrial setup. The task is designed in both virtual and physical environments.

Socially-Assistive Robotics (SAR) has been extensively used for a variety of applications, including educational assistants, exercise coaches and training task instructors. The main goal of such systems is to provide a personalized and tailored session that matches user abilities and needs. While objective measures (e.g., task performance) can be used to adjust task parameters (e.g., task difficulty), towards personalization, it is essential that such systems also monitor task engagement to personalize their training strategies and maximize the effects of the training session. We propose an Interactive Reinforcement Learning (IRL) framework that combines explicit feedback (task performance) with implicit human-generated feedback (task engagement) to achieve efficient personalization. We illustrate the framework with a cognitive training task, describing our data-driven methodology (data collection and analysis, user simulation) towards designing our proposed real-time system. Our data analysis and the reinforcement learning experiments on real user data indicate that the integration of task engagement as human-generated feedback in the RL mechanism can facilitate robot personalization, towards a real-time personalized robot-assisted training system.

As more industries adopt the use of robots to increase productivity, there is an increased need for effective human-robot interaction training, especially in the case of heavy and high precision robots. This implies the need for easy assessment methods that ensure accurate and personalized employee training. Most current assessments are done via manual observation and surveys. This paper addresses the need for the design of intelligent systems to assess a user's training needs based on the user's behavior and engagement while performing a vocational task simulation. In this paper, we propose a multi-sensory intelligent system to predict user engagement using facial expression and body posture data while the user performs a task to provide cognitive assessment of the user's capabilities, a critical factor in successful vocational performance using robots.

Vocational assessment is the process of identifying and assessing an individual’s level of functioning in relation to vocational preparation. In this research, we have designed a framework to evaluate and train the visual working memory and attention level of users by using a humanoid robot and a brain headband sensor. The humanoid robot generates a sequence of colors and the user performs the task by arranging the colored blocks in the same order. In addition, a task-switching paradigm is used to switch between the tasks and colors to give a new instruction to the user by the robot. The humanoid robot displays guidance error detection information, observes the performance of users during the assessment and gives instructive feedback to them. This research describes the profile of cognitive and behavioral characteristics associated with visual working memory skills, selective attention and ways of supporting the learning needs of workers affected by this problem. Finally, the research concludes the relationships between visual working memory and attentional level during different level of the assessment.

Robot Assisted Training (RAT) systems have been successfully deployed to provide assistance during a training task, promoting an efficient interaction with the user. Personalization can improve the efficiency of the interaction and thus enhance the effects of the training session. Personalization can be achieved through user skill assessment in order to choose an appropriate robot behavior that matches user abilities and needs. Graphical User Interfaces have been used to enable human supervisors to control robots and guide the interaction in RAT-based systems. This work focuses on how such interfaces can be used to enable human supervisor users (e.g., therapists) to assess user skills during a robot-based cognitive task. In this study, we investigate how different visualization features affect decision making and efficiency, towards the design of an intelligent and informative interface.

In this paper, we focus on estimating the forward kinematic equation of robots with multilayer feed-forward neural networks. The effectiveness of this approach is tested on a simulated kinematic model of the 7-DOF Sawyer Robotic Arm. In the initial sections of the paper, we discuss related work that associates with the creation of model agnostic control schemes on a kinematic level. Moreover, we formalize the kinematic problem as a supervised problem and we propose an MLP architecture to solve the problem. Lastly, we present experimental results and discuss the potential and importance to create model agnostic control schemes with machine learning.

Smart wearable devices have lead to an increased need for processing and sharing large streams of physiological data in real-time. Modern Human-Machine Interaction (HMI) systems, especially applications designed for user training and assessment (e.g., educational or smart-rehabilitation systems), should be able to track and monitor those signals and adapt their parameters accordingly in order to optimally facilitate the special needs of each individual. Towards this end, we propose a passive Brain-Computer Interface (BCI), using a wireless non-intrusive EEG sensor under a robot assisted training task designed for cognitive assessment. As part of this ongoing work, we demonstrate our initial results on predicting user's task performance, from the EEG signals, before task completion. Our findings highlight the potentials of our hypotheses as we achieve a maximum accuracy rate equal to 74% when evaluated on 69 real subjects.

This paper describes a novel teleoperation interface system to program industrial robotic arms. The system demonstrates the potential to create a programmable interface that enables users with no prior knowledge of robotics to safely program mechanical manipulators with the use of Virtual Reality and the Leap Motion Controller. The system takes full advantage of the Leap Motion to navigate the virtual workspace of the robot that was created through the kinematic properties of the real robot. The implementation of the application was deemed possible by interfacing the Unity Engine with the Barrett WAM robotic arm. Preliminary experimental results show the ability of the system to engage and train appropriately the user in robot programming.

In this paper, we propose an Interactive Learning and Adaptation framework for Human-Robot Interaction in a vocational setting. We show how Interactive Reinforcement Learning (RL) techniques can be applied to such HRI applications in order to promote effective interaction. We present the framework by showing two different use cases in a vocational setting. In the first use case, the robot acts as a trainer, assisting the user while the user is solving the Towers of Hanoi problem. In the second use case, a robot and a human operator collaborate towards solving a synergistic construction or assembly task. We show how RL is used in the proposed framework and discuss its effectiveness in the two different vocational use cases, the Robot Assisted Training and the Human-Robot Collaboration case.

Upper-limb robotic rehabilitation systems should inform the therapists for their patients status. Such therapy systems must be developed carefully by taking into consideration real life uncertainties that associate with sensor error. In our paper, we describe a system which is composed of a depth camera that tracks the motion of the patients upper limb, and a robotic manipulator that challenges the patient with repetitive exercises. The goal of this study is to propose a motion analysis system that improves the readings of the depth camera, through the use of a kinematic model that describes the motion of the human arm. In our current experimental set-up we are using the Kinect v2 to capture a participant who performs rehabilitation exercises with the Barrett WAM robotic manipulator. Finally, we provide a numerical comparison among the stand alone measurements from the Kinect v2, the estimated motion parameters of our system and the VICON, which we consider as an error-free ground truth apparatus.

In this paper, we present an adaptive cognitive music game designed to monitor and improve the attention levels of people with dementia. The goal of this game is to provide a customized protocol based on user needs and preferences, following the Reinforcement Learning (RL) framework. The game adjusts its parameters (e.g., difficulty level) so as to help the user complete the task successfully, while keeping them engaged. The main contribution of this paper is an interactive learning and adaptation framework that enables and facilitates the adaptation of the robot behavior towards new users, providing a safe, tailored and efficient interaction.

In this paper, we present an interactive learning and adaptation framework. The framework combines Interactive Reinforcement Learning methods to effectively adapt and refine a learned policy to cope with new users. We argue that implicit feedback provided by the primary user and guidance from a secondary user can be integrated to the adaptation mechanism, resulting at a tailored and safe interaction. We illustrate this framework with a use case in Robot Assisted Therapy, presenting a Robot Yoga Trainer that monitors a yoga training session, adjusting the session parameters based on human motion activity recognition and evaluation through depth data, to assist the user complete the session, following a Reinforcement Learning approach.

During the last two decades, robotic rehabilitation has become widespread, particularly for upper limb physical rehabilitation. Major findings prove that the efficacy of robot-assisted rehabilitation can be increased by motivation and engagement, which is offered by exploiting the opportunities of gamification and exergaming. This paper presents a tele-rehabilitation framework to enable interaction between therapists and patients and is a combination of a graphical user interface and a high dexterous robotic arm. The system, called MAGNI, integrates a 3D exercise game with a robotic arm, operated by therapist in order to assign in real-time the prerecorded exercises to the patients. We propose a game that can be played by a patient who has suffered an injury to their arm (e.g. Stroke, Spinal Injury, or some physical injury to the shoulder itself). The experimental results and the feedback from the participants show that the system has the potential to impact how robotic physical therapy addresses specific patient’s needs and how occupational therapists assess patient’s progress over time.

In this paper, we present an interactive learning and adaptation framework that facilitates the adaptation of an interactive agent to a new user. We argue that Interactive Reinforcement Learning methods can be utilized and integrated to the adaptation mechanism, enabling the agent to refine its learned policy in order to cope with different users. We illustrate our framework with a use case in the domain of Robot Assisted Therapy. We present our results of the learning and adaptation experiments against different simulated users, showing the motivation of our work and discussing future directions towards the definition and implementation of our proposed framework.

A great deal of recent research has focused on social and assistive robots that can achieve a more natural and realistic interaction between the agent and its environment. Following this direction, this paper aims to establish a computational framework that can associate objects with their uses and their basic characteristics in an automated manner. The goal is to continually enrich the robot's knowledge regarding objects that are important to the user, through verbal interaction. We address the problem of learning correlations between object properties and human needs by associating visual with verbal information. Although the visual information can be acquired directly by the robot, the verbal information is acquired via interaction with a human user. Users provide descriptions of the objects for which the robot has captured visual information, and these two sources of information are combined automatically. We present a general model for learning these associations using Gaussian Mixture Models. Since learning is based on a probabilistic model, the approach handles uncertainty, redundancy, and irrelevant information. We illustrate the capabilities of our approach by presenting the results of an initial experiment run in a laboratory environment, and we describe the set of modules that support the proposed framework.

The estimation of human arm forces is an important factor in physical therapy, especially in robotic-aided physical therapy. Force measurements reveal the rehabilitation progress of patients with poor upper extremity motor function. In this work, we managed to record and analyse the upper arm forces of the patient while executing upper extremity exercises. Our analysis of these forces allows the robot to identify the patient's motion capability and apply active motion control to the patient's arm when their arm's forces deviate from the desired motion set-up by the therapist.

In this paper, we present a framework for physical rehabilitation, that uses a combination of video gaming and robotic technology to allow the monitoring and progress tracking of a person during physical therapy. The system, called MAGNI, uses the advanced control capabilities of the Barrett WAM Arm robot and a custom-made video game. The MAGNI system helps the patient to complete a rehabilitation session through a user-system, game-based interaction program, involving exercises prescribed by a therapist. The system can control and supervise the rehabilitation sessions to ensure compliance and safe exercising. It uses motion analysis to provide an evaluation of the patient's progress over time. The MAGNI system records the position of the subject's hand during game interaction with the robotic arm and analyzes this data using pattern matching and machine learning algorithms, in order to guide self-managed physical therapy. Our experiments show that we can accurately classify user motion activity between a set of different exercises, and measure user compliance with the prescribed regimens.

This paper presents an Unmanned Aerial Vehicle (UAV), based on the AR.Drone platform, which can perform an autonomous navigation in indoor (e.g. corridor, hallway) and industrial environments (e.g. production line). It also has the ability to avoid pedestrians while they are working or walking in the vicinity of the robot. The only sensor in our system is the front camera. For the navigation part our system rely on the vanishing point algorithm, the Hough transform for the wall detection and avoidance, and the HOG descriptors for pedestrian detection using SVM classifier. Our experiments show that our vision navigation procedures are reliable and enable the aerial vehicle to fly without humans intervention and coordinate together in the same workspace. We are able to detect human motion with high confidence of 85% in a corridor and to confirm our algorithm in 80% successful flight experiments.

In this paper, we present an ongoing effort to develop a robust omnidirectional robotic platform for outdoor operation on non-smooth surfaces. The design of an off-road, low-cost omniwheel is presented along with a suspension system that will allow the platform to traverse rough terrain. We also provide a control architecture based on the open-source Robotic Operating System (ROS).

In this paper, we present a system targeted for assistive living environments, that is able to control the actuators of a robot and manipulate objects that lie on a table. An adaptive dialogue system is responsible for interacting with the user, retrieving his/her intentions and reacting accordingly. Our system is able to learn in real time how to solve complicated tasks by combining solutions to simpler ones, thus reusing previous knowledge. It is also able to effectively plan and execute table top object manipulation, which can have a great impact in the quality of life of an elderly, disabled or injured person.

Context-awareness constitutes a fundamental attribute of a smart environment. Our research aims at advancing the context-awareness capabilities of ambient intelligence environments by combining multi-modal information from both stationary and moving sensors. The collected data enables us to perform person identification and 3-D localization and recognize activities. In addition, we explore closed-loop feedback by integrating autonomous robots interacting with the users.

We present the development of a novel system that optimally positions a number of robots in an assistive environment and can be used in a domestic environment inhabited by an elderly or disabled person, for event recognition. Placement of the robots is done by means of the Extended Max Sum Decentralized Coordination algorithm. The robot used is based on the iRobot Create platform that is controlled by a laptop and carries a number of multimodal sensors. The sensor readings are taken into account during the optimal placement process, in addition to the environment layout, in order to maximize the system's effectiveness.

Social robotics has emerged as a new research area in recent years. One of the reasons behind this emergence is the rapid pace of improvements in sensor, actuator and processing capabilities in modern hardware enabling robots to interact with humans more effectively than ever before. The motivation for the work presented in this paper is to use advanced human-robot head-eye interaction algorithms in order to create a robotic framework that assists physical therapists treating sensor-motor impairments, such as Autism and Cerebral Palsy by using robotic systems. The robotic platform used in our work is the social robot Zeno, which has a fantastically friendly appearance and bridges the previously reported uncanny valley. In this paper we report on a new coordination algorithm based on reinforcement learning implemented on Zeno for achieving human like head-eye coordination to visually engage patients with cognitive impairments. The experimental results show that the various methods implemented enables social robot Zeno achieve natural head-eye coordination with significant improvement in accuracy without the need of extensive kinematic analysis of the system.

Visually impaired people can navigate unfamiliar areas by relying on the assistance of other people, canes, or specially trained guide dogs. Guide dogs provide the impaired person with the highest degree of mobility and independence, but require expensive training and selective breeding. In this paper we describe the design and development of a prototype assistive-guide robot (eyeDog) that provides the visually impaired person with autonomous vision-based navigation and laser-based obstacle avoidance capabilities. This kind of assistive-guide robot has several advantages, such as robust performance and reduced cost and maintenance. The main components of our system are the Create robotic platform (from iRobot), a net-book, an on-board USB webcam and a LIDAR unit. The camera is used as the primary exteroceptive sensor for the navigation task; the frames captured by the camera are processed in order to robustly estimate the position of the vanishing point associated to the road/corridor where the eyeDog needs to move. The controller will then steer the robot until the vanishing point and the image center coincide. This condition guarantees the robot to move parallel to the direction of the road/corridor. While moving, the robot uses the LIDAR for obstacle avoidance.

This paper describes Neptune, a mobile manipulator designed as an assistive device for the rehabilitation of children with special needs, such as those suffering from Cerebral-Palsy. Neptune consists of a mobile robot base and a 6DOF robotic arm, and it is interfaced to users via Wii Remote, iPad, Neural Headset, a camera, and pressure sensors. These interfaces allow patients, therapists and operators to interact with the robot in multiple ways, as may be appropriate in assistive scenarios such as: direct physical interaction with the iPad, arm positioning exercises through WiiMote, remote navigation and object retrieval through the environment via the Neural Headset, etc. In this paper we present an overview of the system and discuss its future uses in rehabilitation of CP children.

Everyone is familiar with the scenario, people demand or assign tasks to robots, and robots execute the tasks to serve people. We call such a model Serve-on-Demand. With the advancement of pervasive computing, machine learning and artificial intelligence, the robot service of the next generation will inevitably turn to actively and exactly meet people’s needs, even without explicit demand. We call it Serve-on-Need. It requires the robots to comprehend the intentions and preferences of people exactly. In this paper, we model the human-computer interaction for Serve-on-Need as a repeated stochastic Bayesian game. We solve the stochastic Bayesian game by an equilibrium analysis and rational learning. We present the service of a coffee robot to illustrate such an approach.

Explicit communication addresses the use of distinct language or protocol to convey the idea. Implicit communication helps to compensate many hidden meanings omitted from the explicit language. In some situations, implicit communication may even take the place of explicit communication. For the autonomous robot, implicit communication provides an alternative way to interact with people. This paper introduces the acoustic techniques for implicit communication in human-robot interaction, and the design of acoustical implicit communication based robot games.

An assistive environment is a smart domestic space based on pervasive computing to support the elderly and disabled. Unlike sensors, which can only provide passive monitoring, a robot can be an active element to improve the quality of life for the human. In this paper, we propose an active service of the robot in assistive environments, to help human in the case of emergency situation. It works on a hierarchical partially observable Markov decision process (POMDP). The multimodal observation series are used in the decision and evaluation process.An active robot is a kind of robot that can provide a preferable and necessary active service to the human. This is used in our emergency response system (ERS) to deal with the emergency situations, such as an older adult falls down or emergency diseases. The purpose of multimodal observations is to guarantee the precision of report for the emergency situations. Four observation sources are introduced in this paper: the vision recognition, the voice recognition, the physical input devices and the foreign systems. For each observation source, there are two observation series. Multiple information sources give the agent more opportunities to learn from the real world, so as to make more reasonable predictions, evaluations and decisions.

This work describes a new method for locating moving lips in digital video sequences and applications for this technology in multimedia information retrieval and robotics. The capture on digital video of groups of speakers is very common. Determining which person in such a group is speaking is useful in many applications, including robot navigation, speaker identification and video segmentation for the purpose of retrieval in recorded proceedings.