Search

In Human Robot Interaction (HRI) scenarios, robot systems would benefit from an understanding of the user's state, actions and their effects on the environments to enable better interactions. While there are specialised vision algorithms for different perceptual channels, such as objects, scenes, human pose, and human actions, it is worth considering how their interaction can help improve each other's output. In computer vision, individual prediction modules for these perceptual channels frequently produce noisy outputs due to the limited datasets used for training and the compartmentalisation of the perceptual channels, often resulting in noisy or unstable prediction outcomes. To stabilise vision prediction results in HRI, this paper presents a novel message passing framework that uses the memory of individual modules to correct each other's outputs. The proposed framework is designed utilising common-sense rules of physics (such as the law of gravity) to reduce noise while introducing a pipeline that helps to effectively improve the output of each other's modules. The proposed framework aims to analyse primitive human activities such as grasping an object in a video captured from the perspective of a robot. Experimental results show that the proposed framework significantly reduces the output noise of individual modules compared to the case of running independently. This pipeline can be used to measure human reactions when interacting with a robot in various HRI scenarios.

The goal of optimization-based meta-learning is to find a single initialization shared across a distribution of tasks to speed up the process of learning new tasks. Conditional meta-learning seeks task-specific initialization to better capture complex task distributions and improve performance. However, many existing conditional methods are difficult to generalize and lack theoretical guarantees. In this work, we propose a new perspective on conditional meta-learning via structured prediction. We derive task-adaptive structured meta-learning (TASML), a principled framework that yields task-specific objective functions by weighing meta-training data on target tasks. Our non-parametric approach is model-agnostic and can be combined with existing meta-learning methods to achieve conditioning. Empirically, we show that TASML improves the performance of existing meta-learning models, and outperforms the state-of-the-art on benchmark datasets.

Social or humanoid robots do hardly show up in "the wild", aiming at pervasive human benefits such as child health. This paper presents the socio-cognitive engineering (SCE) methodology for the required field research \& development of robots, focusing on the incremental development of a social robot and child-robot activities that support the daily diabetes management processes of children, aged between 7 and 14 years (i.e., supporting a healthy lifestyle). The SCE methodology helps to integrate into the human-agent/robot system: (a) theories, models and methods from different scientific disciplines, (b) technologies from different fields, (c) varying diabetes management practices, and (d) last but not least, the diverse individual and context-dependent needs of the patients and caregivers. The resulting system represents a new type of long-term human-robot partnerships with evolving collective intelligence. The current prototype is based on four human-robot partnership functions, a knowledge-base and interaction design for child's prolonged disease self-management. It has been developed and tested in three cycles, and proved to support the children on the three basic needs of the Self-Determination Theory: autonomy, competence and relatedness. To our knowledge, this is the first design \& test of a robot system "in the wild" for prolonged "blended" care of children with a chronic disease, showing positive results in a 3 month evaluation period.

In this paper we present a novel augmented reality head mounted display user interface for controlling a robotic wheelchair for people with limited mobility. To lower the cognitive requirements needed to control the wheelchair, we propose integration of a smart wheelchair with an eye-tracking enabled head-mounted display. We propose a novel platform that integrates multiple user interface interaction methods for aiming at and selecting affordances derived by on-board perception capabilities such as laser-scanner readings and cameras. We demonstrate the effectiveness of the approach by evaluating our platform in two realistic scenarios: 1) Door detection, where the affordance corresponds to a Door object and the Go-Through action and 2) People detection, where the affordance corresponds to a Person and the Approach action. To the best of our knowledge, this is the first demonstration of a augmented reality head-mounted display user interface for controlling a smart wheelchair.

This paper addresses the problem of motion segmentation of articulated rigid bodies from a single-view RGB-D data sequence. Current methods either perform dense motion segmentation, and consequently are very computational demanding, or rely on sparse 2D feature points, which may not be sufficient to represent the entire scene. In this paper, we advocate the use of 3D semi-dense motion segmentation which also bridges some limitations of standard 2D methods ( e.g . background removal). We cast the 3D motion segmentation problem into a subspace clustering problem, adding an adaptive spectral clustering that estimates the number of object rigid parts. The resultant method has few parameters to adjust, takes less time than the temporal length of the scene and requires no post-processing.

To understand people and their intentions, humans have developed the ability to imagine their surroundings from another visual point of view. This cognitive ability is called perspective taking and has been shown to be essential in child development and social interactions. However, the precise cognitive mechanisms underlying perspective taking remain to be fully understood. Here we present a computa- tional model that implements perspective taking as a mental simulation of the physical movements required to step into the other point of view. The visual percept after each mental simulation step is estimated using a set of forward models. Based on our experimental results, we propose that a visual attention mechanism explains the response times reported in human visual perspective taking experiments. The model is also able to generate several testable predictions to be explored in further neurophysiological studies.

Generating complex, human-like behavior in a humanoid robot like the iCub requires the integration of a wide range of open source components and a scalable cognitive architecture. Hence, we present the iCub-HRI library which provides convenience wrappers for components related to perception (object recognition, agent tracking, speech recognition, and touch detection), object manipulation (basic and complex motor actions), and social interaction (speech synthesis and joint attention) exposed as a C++ library with bindings for Java (allowing to use iCub-HRI within Matlab) and Python. In addition to previously integrated components, the library allows for simple extension to new components and rapid prototyping by adapting to changes in interfaces between components. We also provide a set of modules which make use of the library, such as a high-level knowledge acquisition module and an action recognition module. The proposed architecture has been successfully employed for a complex human–robot interaction scenario involving the acquisition of language capabilities, execution of goal-oriented behavior and expression of a verbal narrative of the robot’s experience in the world. Accompanying this paper is a tutorial which allows a subset of this interaction to be reproduced. The architecture is aimed at researchers familiarizing themselves with the iCub ecosystem, as well as expert users, and we expect the library to be widely used in the iCub community. The research leading to these results has received funding under the European Union’s Seventh Framework Programme (FP/2007-2013)/ERC Grant Agreement n. FP7-ICT-612139 (WYSIWYD—What You Say Is What You Did) and FP7-ICT-270490 (EFAA—The Experimental Functional Android Assistant). PN was supported by a Marie Curie Early Stage Researcher Fellowship (H2020-MSCA-ITA, SECURE 642667). PV was supported by the ERC advanced grant 341196 (cDAC—Role of Consciousness in Adaptive Behavior).

Various research topics are emerging as the demand for intelligent lifelong interactions between robot and humans increases. Among them, we can find the examination of persistent storage, the continuous unsupervised annotation of memories and the usage of data at high-frequency over long periods of time. We recently proposed a lifelong autobiographical memory architecture tackling some of these challenges, allowing the iCub humanoid robot to 1) create new memories for both actions that are self-executed and observed from humans, 2) continuously annotate these actions in an unsupervised manner, and 3) use reasoning modules to augment these memories a-posteriori. In this paper, we present a reasoning algorithm which generalises the robots’ understanding of actions by finding the point of commonalities with the former ones. In particular, we generated and labelled templates of pointing actions in different directions. This represents a first step towards the emergence of a procedural memory within a long-term autobiographical memory framework for robots.

In the future, robots will support humans in their every day activities. One particular challenge that robots will face is understanding and reasoning about the actions of other agents in order to cooperate effectively with humans. We propose to tackle this using a developmental framework, where the robot incrementally acquires knowledge, and in particular 1) self-learns a mapping between motor commands and sensory consequences, 2) rapidly acquires primitives and complex actions by verbal descriptions and instructions from a human partner, 3) discovers correspondences between the robots body and other articulated objects and agents, and 4) employs these correspondences to transfer the knowledge acquired from the robots point of view to the viewpoint of the other agent. We show that our approach requires very little a-priori knowledge to achieve imitation learning, to find correspondent body parts of humans, and allows taking the perspective of another agent. This represents a step towards the emergence of a mirror neuron like system based on self-learned representations.

In this paper, we present an approach to enable a humanoid robot to provide personalised dressing assistance for human users using multi-modal information. A depth sensor is mounted on top of the robot to provide visual information, and the robot end effectors are equipped with force sensors to provide haptic information. We use visual information to model the movement range of human upper-body parts. The robot plans the dressing motions using the movement range models and real-time human pose. During assistive dressing, the force sensors are used to detect external force resistances. We present how the robot locally adjusts its motions based on the detected forces. In the experiments we show that the robot can assist human to wear a sleeveless jacket while reacting to the force resistances.

Forward models play a key role in cognitive agents by providing predictions of the sensory consequences of motor commands, also known as sensorimotor contingencies (SMCs). In continuously evolving environments, the ability to anticipate is fundamental in distinguishing cognitive from reactive agents, and it is particularly relevant for autonomous robots, that must be able to adapt their models in an online manner. Online learning skills, high accuracy of the forward models and multiple-step-ahead predictions are needed to enhance the robots’ anticipation capabilities. We propose an online heterogeneous ensemble learning method for building accurate forward models of SMCs relating motor commands to effects in robots’ sensorimotor system, in particular considering proprioception and vision. Our method achieves up to 98% higher accuracy both in short and long term predictions, compared to single predictors and other online and offline homogeneous ensembles. This method is validated on two different humanoid robots, namely the iCub and the Baxter.

Standard powered wheelchairs are still heavily dependent on the cognitive capabilities of users. Unfortunately, this excludes disabled users who lack the required problem-solving and spatial skills, particularly young children. For these children to be denied powered mobility is a crucial set-back; exploration is important for their cognitive, emotional and psychosocial development. In this paper, we present a safer paediatric wheelchair: the Assistive Robot Transport for Youngsters (ARTY). The fundamental goal of this research is to provide a key-enabling technology to young children who would otherwise be unable to navigate independently in their environment. In addition to the technical details of our smart wheelchair, we present user-trials with able-bodied individuals as well as one 5-year-old child with special needs. ARTY promises to provide young children with early access to the path towards mobility independence.

Independent mobility is crucial for a growing child and its loss can severely impact cognitive, emotional and social development. Unfortunately, powered wheelchair provision for young children has been difficult due to safety concerns. But powered mobility need not be unsafe. Risks can be reduced through the use of robotic technology (e.g., obstacle avoidance) and we present a prototype safe smart paediatric wheelchair: the Assistive Robot Transport for Youngsters (ARTY). A core aspect of our work is that we aim to bring ARTY to the field and we discuss the challenges faced when trying to involve children in the development/testing of medical technology. We discuss one preliminary experiment designed as a “Hide-and-Seek” game as a short case study.

Artificial companion agents have the potential to combine novel means for effective health communication with young patients support and entertainment. However, the theory and practice of long-term child-robot interaction is currently an underdeveloped area of research. This paper introduces an approach that integrates multiple functional aspects necessary to implement temporally extended human-robot interaction in the setting of a paediatric ward. We present our methodology for the implementation of a companion robot which will be used to support young patients in hospital as they learn to manage a lifelong metabolic disorder (diabetes). The robot will interact with patients over an extended period of time. The necessary functional aspects are identified and introduced, and a review of the technical challenges involved is presented.

A wide range of architectures have been proposed within the areas of learning by demonstration and multi-robot coordination. These areas share a common issue: how humans and robots share information and knowledge among themselves. This paper analyses the impact of communication between human teachers during simultaneous demonstration of task execution in the novel Multi-robot Learning by Demonstration domain, using the MRLbD architecture. The performance is analysed in terms of time to task completion, as well as the quality of the multi-robot joint action plans. Participants with different levels of skills taught real robots solutions for a furniture moving task through teleoperation. The experimental results provided evidence that explicit communication between teachers does not necessarily reduce the time to complete a task, but contributes to the synchronisation of manoeuvres, thus enhancing the quality of the joint action plans generated by the MRLbD architecture.

Teaching a robot to learn new knowledge is a repetitive and tedious process. In order to accelerate the process, we propose a novel template-based approach for robot arm movement imitation. This algorithm selects a previously observed path demonstrated by a human and generates a path in a novel situation based on pairwise mapping of invariant feature locations present in both the demonstrated and the new scenes using a combination of minimum distortion and minimum energy strategies. This One-Shot Learning algorithm is capable of not only mapping simple point-to-point paths but also adapting to more complex tasks such as those involving forced waypoints. As compared to traditional methodologies, our work require neither extensive training for generalisation nor expensive run-time computation for accuracy. This algorithm has been statistically validated using cross-validation of grasping experiments as well as tested for practical implementation on the iCub humanoid robot for playing the tic-tac-toe game.

One of the most important aspects of a compelling game AI is that it anticipates the player’s actions and responds to them in a convincing manner. The first step towards doing this is to understand what the player is doing and predict their possible future actions. In this paper we show an approach where the AI system focusses on testing hypotheses made about the player’s actions using an implementation of a cognitive architecture inspired by the simulation theory of mind. The application used in this paper is to predict the target that the player is heading towards, in an RTS-style game. We improve the prediction accuracy and reduce the number of hypotheses needed by using path planning and path clustering.

Collaborative control architectures assist human users in performing tasks, without undermining their capabilities or curtailing the natural development of their skills. In this study, we evaluate our collaborative control architecture by investigating the visual attention patterns of robotic wheelchair users. Our initial hypothesis stated that the user would require less visual attention for driving, whilst they are being assisted by the collaborative system, thus allowing them to concentrate on higher level cognitive tasks, such as planning. However, our analysis of eye gaze patterns—as recorded by a head mounted eye tracking system—supports the opposite conclusion: that patterns of saccadic activation increase and become more chaotic under the assisted mode. Our findings highlight the necessity for techniques that assist the user in forming an appropriate mental model of the collaborative control architecture.

We examine the application of spectral clustering for breaking up the behaviour of a multi-agent system in space and time into smaller, independent elements. We extend the clustering into the temporal domain and propose a novel similarity measure, which is shown to possess desirable temporal properties when clustering multi-agent behaviour. We also propose a technique to add knowledge about events of multi-agent interaction with different importance. We apply spectral clustering with this measure for analysing behaviour in a strategic game.

Current music production software allows rapid programming of drum patterns, but programmed patterns often lack the groove that a human drummer will provide, both in terms of being rhythmically too rigid and having no variation for longer periods of time. We have implemented an artificial software drummer that learns drum patterns by extracting user specific variations played by a human drummer. The artificial drummer then builds up a library of patterns it can use in different musical contexts. The artificial drummer models the groove and the variations of the human drummer, enhancing the realism of the produced patterns.

Many excellent architectures exist that allow imitation of actions involving observable goals. In this paper, we develop a Simulation Theory-based architecture that uses continuous visual perspective taking to maintain a persistent model of the demonstrators knowledge of object locations in dynamic environments; this allows an observer robot to attribute potential actions in the presence of goal occlusions, and predict the unfolding of actions through prediction of visual feedback to the demonstrator. The architecture is tested in robotic experiments, and results show that the approach also allows an observer robot to solve Theory-of-Mind tasks from the False Belief paradigm.

We present an architecture that enables asocial and social learning mechanisms to be combined in a unified framework on a robot. The robot learns two kinds of internal models by interacting with the environment with no a priori knowledge of its own motor system: internal object models are learnt about how its motor system and other objects appear in its sensor data; internal control models are learnt by babbling and represent how the robot controls objects. These asocially-learnt models of the robot’s motor system are used to understand the actions of a human demonstrator on objects that they can both interact with. Knowledge acquired through self-exploration is therefore used as a bootstrapping mechanism to understand others and benefit from their knowledge.

In this study, we investigated the use of van der Pol oscillators in a 4-dof embodied bipedal robotic platform for the purposes of planar walking. The oscillator controlled the hip and knee joints of the robot and was capable of generating waveforms with the correct frequency and phase so as to entrain with the mechanical system. Lowering its oscillation frequency resulted in an increase to the walking pace, indicating exploitation of the global natural dynamics. This is verified by its operation in absence of entrainment, where faster limb motion results in a slower overall walking pace.

Forward models enable both robots and humans to predict the sensory consequences of their motor actions. To learn its own forward models a robot needs to experiment with its own motor system, in the same way that human infants need to babble as a part of their motor development. In this paper we investigate how this babbling with the motor system can be influenced by the forward models’ own knowledge of their predictive ability. By spending more time babbling in regions of motor space that require more accuracy in the forward model, the learning time can be reduced. The key to guiding this exploration is the use of probabilistic forward models, which are capable of learning and predicting not just the sensory consequence of a motor command, but also an estimate of how accurate this prediction is. An experiment was carried out to test this theory on a robotic pan tilt camera.

In this paper, the effects of different lower-level building blocks of a robotic swinging system are explored, from the perspective of motor skill acquisition. The van der Pol and Rayleigh oscillators are used to entrain to the system’s natural dynamics, with two different network topologies being used: a symmetric and a hierarchical one. Rayleigh outperformed van der Pol regarding maximum oscillation amplitudes for every morphological configuration examined. However, van der Pol started large amplitude relaxation oscillations faster, attaining better performance during the first half of the transient period. Hence, even though there are great similarities between the oscillators, differences in their resultant behaviours are more pronounced than originally expected.

In this study, we investigated the use of van der Pol oscillators in a 2-dof embodied robotic platform for a swinging task. The oscillator controlled the hip and knee joints of the robot and was capable of generating waveforms with the correct frequency and phase so as to entrain with the mechanical system.

Coupling internal inverse and forward models gives rise to on-line simulation processes that may be used as a common computational substrate for action execution, planning, recognition, imitation and learning. In this paper, multiple coupled internal inverse and forward models are arranged in a hierarchical fashion, with each level of the hierarchy interacting with other levels through top-down and bottom-up processes. Through experiments involving imitation of a human demonstrator performing object manipulation tasks, this architecture is shown to equip a robot with a multi-level motor abstraction capability. This is then used to solve the correspondence problem in action recognition. The architecture is inspired by biological evidence.

A considerable part of the imitation problem is finding mechanisms that link the recognition of actions that are being demonstrated to the execution of the same actions by the imitator. In a situation where a human is instructing a robot, the problem is made more complicated by the difference in morphology. In this paper we present an imitation framework that allows a robot to recognise and imitate object-directed actions performed by a human demonstrator by solving the correspondence problem. The recognition is achieved using an abstraction mechanism that focuses on the features of the demonstration that are important to the imitator. The abstraction mechanism is applied to experimental scenarios in which a robot imitates human- demonstrated tasks of transporting objects be- tween tables.

For a robot to successfully learn from demonstration it must posses the ability to reproduce the actions of a teacher. For this to happen, the robot must generate motor signals to match its proprioceptively perceived state with that of the visually perceived state of a teacher. In this paper we describe a real time matching model at a neural level of description. Experimental results from matching of arm movements, using dynamically simulated articulated robots, are presented.