Search

Humans can perform object manipulations in VR in spite of missing haptic and acoustic information. Whether their movements under these artificial conditions do still rely on motor programs based on natural experience or are impoverished due to the restrictions imposed by VR is unclear. We investigated whether reach-to-place and reach-to-grasp movements in VR can still be adapted to the task and to the specific properties of the objects being handled, or whether they reflect a stereotypic, task- and object-independent motor program. We analyzed reach-to-grasp and reach-to-place movements from participants performing an unconstrained ”set-the-table” task involving a variety of different objects in virtual reality. These actions were compared based on their kinematic features. We encountered significant differences in peak speed and the duration of the deceleration phase which are modulated depending on the action and on the manipulated object. The flexibility of natural human sensorimotor control thus is at least partially transferred and exploited in impoverished VR conditions. We discuss possible explanations of this behavior and the implications for the design of object manipulations in VR.

Combining machine learning in neural networks with multimodal fusion strategies offers an interesting potential for classification tasks but the optimum fusion strategies for many applications have yet to be determined. Here we address this issue in the context of human activity recognition, making use of a state-of-the-art convolutional network architecture (Inception I3D) and a huge dataset (NTU RGB+D). As modalities we consider RGB video, optical flow, and skeleton data. We determine whether the fusion of different modalities can provide an advantage as compared to uni-modal approaches, and whether a more complex early fusion strategy can outperform the simpler late-fusion strategy by making use of statistical correlations between the different modalities. Our results show a clear performance improvement by multi-modal fusion and a substantial advantage of an early fusion strategy

Contact events during manipulation tasks can be distinguished in two categories: intended and unintended. We investigated the categorization of contact events during an object placing task executed in virtual reality based on kinematic features measured during the movement segment previous to the contact. The experimental setup enabled us to generate unintended contacts by triggering unexpected interruptions of the placing movement. Experimental results indicate that the kinematic features enable the distinction of intended and unintended contacts independent of substantial variations of movement properties (amplitude, duration, velocity), unless the unintended contact occurs toward the end of the planned movement.

Contact events are essential to the execution of manipulation tasks in virtual reality because they mark the different phases of an interaction, e.g. initial hand-object or tool-object contact. Generally, the nature of contact events can be distinguished in two categories: intended and unintended. Whereas intended contacts typically signalize the successful execution of a task, unintended contacts are the result of bad motor planning or execution, or are caused by accidental movements.Here we explore the use of kinematic features to distinguish intended and unintended contacts during an object placing task executed in virtual reality. A handheld object was placed over a horizontal surface with a vertical planar movement. The experimental setup enabled us to generate unintended contacts by triggering unexpected interruptions during the placing movement.Experimental results show significant differences between distributions of the kinematic features across intended and unintended contacts. These results indicate that the proposed kinematic features enable a robust distinction between intended and unintended contacts independent of substantial variations of movement properties (amplitude, duration, velocity).

The Everyday Activities Science and Engineering (EASE) Collaborative Research Consortium’s mission to enhance the performance of cognition-enabled robots establishes its foundation in the EASE Human Activities Data Analysis Pipeline. Through collection of diverse human activity information resources, enrichment with contextually relevant annotations, and subsequent multimodal analysis of the combined data sources, the pipeline described will provide a rich resource for robot planning researchers, through incorporation in the OpenEASE cloud platform.