Search

Supervising and controlling remote robot systems currently requires many specialised operators to have knowledge of the internal state of the system in addition to the environment. For applications such as remote maintenance of future nuclear fusion reactors, the number of robots (and hence supervisors) required to maintain or decommission a facility is too large to be financially feasible. To address this issue, this work explores the idea of intelligently filtering information so that a single user can supervise multiple robots safely. We gathered feedback from participants using five methods for teleoperating a semi-autonomous multi-robot system via Virtual Reality (VR). We present a novel 3D interaction method to filter the displayed information to allow the user to read information from the environment without being overwhelmed. The novelty of the interface design is the link between Semantic and Spatial filtering and the hierarchical information contained within the multi robot system. We conducted a user study including a cohort of expert robot teleoperators comparing these methods; highlighting the significant effects of 3D interface design on the performance and perceived workload of a user teleoperating many robot agents in complex environments. The results from this experiment and subjective user feedback will inform future investigations that build upon this initial work.

To a great extent, immersion of a virtual environment (VE) depends on the naturalness of the interface it provides for interaction. As people commonly exploit gestures during communication, therefore interaction based on hand-postures enhances the degree of realism of a VE. However, the choice of selecting hand postures for interaction varies from person to person. Generalizing the use of a specific posture with a particular interaction requires considerable computation which in turns depletes intuition of a 3D interface. By investigating machine learning in the domain of virtual reality (VR), this paper presents an open posture-based approach for 3D interaction. The technique is user-independent and relies neither on the size and color of hand nor on the distance between camera and posing-position. The system works in two phases—in the first phase, hand-postures are learnt, whereas in the second phase the known postures are used to perform interaction. With an ordinary camera, a scanned image is partitioned into equal size non-overlapping tiles. Four light-weight features, based on binary histogram and invariant moments, are calculated for each part and portion of a posture-image. The support vector machine classifier is trained by posture-specific knowledge carried accumulatively in each tile. By posing any known posture, the system extracts the tiles information to detect a particular hand-posture. At the successful recognition, appropriate interaction is activated in the designed VE. The proposed system is implemented in a case-study application; vision-based open posture interaction using the libraries of OpenCV and OpenGL. The system is assessed in three separate evaluation sessions. Results of the evaluations testify efficacy of the approach in various VR applications.

Similarity measure of cross-domain descriptors (2D descriptors and 3D descriptors) between 2D image patches and 3D point cloud volumes provides stable retrieval performance and establishes the spatial relationship between 2D and 3D space, which plays the potential applications in geospatial space, such as 2D and 3D interaction of remote sensing, Augmented Reality (AR) and robot navigation. However, the mature handcrafted descriptors of 2D image patches and 3D point cloud volumes are extremely different, resulting in the huge challenge for 2D image patch and 3D point cloud volume matching. In this paper, we propose a novel network which combines both unified descriptor training and descriptor comparison function training for 2D image patch and 3D point cloud volume matching. First, two feature extraction networks are applied for jointly learning the local descriptors for 2D image patches and 3D point cloud volumes, respectively. Second, a fully connected network is introduced to compute the similarity between 2D descriptors and 3D descriptors. Motivated by the successful indicator system on evaluating 2D patch feature representation, we use the false positive rate at 95% recall (FPR95) and precision based on cross-domain descriptors as the measured metric. The experimental results show that our proposed network achieve state-of-the-art performance in the matching of 2D image patches and 3D point cloud volumes.

Precise craniotomy localization is essential in neurosurgical procedures, especially during the preoperative planning. The mainstream craniotomy localization method utilizing image-guided neurosurgery system (IGNS) or augmented reality (AR) navigation system require experienced neurosurgeons to point out the lesion margin by probe and draw the craniotomy manually on the patient's head according to cranial anatomy. However, improper manual operation and dither from the AR model will bring in errors about craniotomy localization. In addition, there is no specific standard to evaluate the accuracy of craniotomy. This paper attempts to propose a standardized interactive 3D method using orthogonal transformation to map the lesion onto the scalp model and generate a conformal virtual incision in real time. Considering clinical requirements, the incision can be amended by 3D interaction and margin modification. According to the IGNS and the virtual incision, an actual craniotomy will be located on the patient's head and the movement path of the probe will be recorded and evaluated by an indicator, which is presented as an evaluated standard to measure the error between virtual and actual craniotomies. After the experiment, an incision is drawn on a 3D printing phantom based on the generated virtual one. The results show that the proposed method can generate a lesion-consistent craniotomy according to the size of the lesion and the mapping angle and delineate the incision on the patient's head precisely under the IGNS.

Eye gaze and head movement are attractive for hands-free 3D interaction in head-mounted displays, but existing interfaces afford only limited control. Radi-Eye is a novel pop-up radial interface designed to maximise expressiveness with input from only the eyes and head. Radi-Eye provides widgets for discrete and continuous input and scales to support larger feature sets. Widgets can be selected with Look & Cross, using gaze for pre-selection followed by head-crossing as trigger and for manipulation. The technique leverages natural eye-head coordination where eye and head move at an offset unless explicitly brought into alignment, enabling interaction without risk of unintended input. We explore Radi-Eye in three augmented and virtual reality applications, and evaluate the effect of radial interface scale and orientation on performance with Look & Cross. The results show that Radi-Eye provides users with fast and accurate input while opening up a new design space for hands-free fluid interaction.

Many interaction techniques in virtual reality break with the 1-to-1 mapping from real to virtual space. Instead, specialized techniques for 3D interaction and haptic retargeting leverage hand redirection, offsetting the virtual hand rendering from the real hand position. To achieve unnoticeable hand redirection, however, the utilization of change blindness phenomena has not been systematically explored. Inspired by recent advances in the domain of redirected walking, we present the first hand redirection technique that makes use of blink-induced visual suppression and corresponding change blindness. We introduce Blink-Suppressed Hand Redirection (BSHR) to study the feasibility and detectability of hand redirection based on blink suppression. Our technique is based on Cheng et al.'s (2017) [9] body warping algorithm and instantaneously shifts the virtual hand when the user's vision is suppressed during a blink. Additionally, it can be configured to continuously increment hand offsets when the user's eyes are opened, limited to an extent below detection thresholds. In a psychophysical experiment, we verify that unnoticeable blink-suppressed hand redirection is possible even in worst -case scenarios, and derive the corresponding conservative detection thresholds (CDTs). Moreover, our results show that the range of unnoticeable redirection can be increased by combining continuous warping and blink-suppressed instantaneous shifts. As an additional contribution, we derive the CDTs for Cheng et al.'s (2017) [9] redirection technique that does not leverage blinks.

Museums and cultural institutions, in general, are in a constant challenge of adding more value to their collections. The attractiveness of assets is practically tightly related to their value obeying the offer and demand law. New digital visualization technologies are found to give more excitements, especially to the younger generation as it is proven by multiple studies. Nowadays, museums around the world are currently trying to promote their collections through new multimedia and digital technologies such as 3D modeling, virtual reality (VR), augmented reality (AR), and serious games. However, the difficulty and the resources required to implement such technologies present a real challenge. Through this paper, we propose a 3D acquisition and visualization framework aiming mostly at increasing the value of cultural collections. This framework preserves cost-effectiveness and time constraints while still introducing new ways of visualization and interaction with high-quality 3D models of cultural objects. The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd 2021. Acknowledgements. This publication was made possible by NPRP grant 9-181-1-036 from the Qatar National Research Fund (a member of Qatar Foundation). The statements made herein are solely the responsibility of the authors (www.ceproqha.qa). Scopus

We demonstrate a mutual telepresence system with a workspace that can cover the whole upper body. A pair of micromirror arrays produces a high-fidelity light field of the other party and the clothes worn by the users provide haptic feedbacks in the interaction. In the demonstration system, we use a technique that broadens the view angle of the 3D optical images by introducing a symmetric mirror structure, where users can see both their body and the partner’s mid-air image simultaneously. The tactile sensation is presented by a jacket-type haptic device in synchronization with the contact with the optical image.

The interaction with distant displays often demands complex, multi-modal inputs which need to be achieved with a very simple hardware solution so that users can perform rich inputs wherever they encounter a distant display. We present Simo, a novel approach, that transforms a regular smartphone into a highly-expressive user motion tracking device and controller for distant displays. Both the front and back cameras of the smartphone are used simultaneously to track the user's hand as well as the head, and body movements in real-world space and scale. In this work, we first define the possibilities for simultaneous face- and world-tracking using current off-the-shelf smartphones. Next, we present the implementation of a smartphone app enabling hand, head, and body motion tracking. Finally, we present a technical analysis outlining the possible tracking range.

This paper proposes a novel concept of a hybrid tactile display with multistimulus feedback, allowing the real-time experience of the position, shape, and texture of the virtual object. The key technology of the TeslaMirror is that we can deliver the sensation of object parameters (pressure, vibration, and electrotactile feedback) without any wearable haptic devices. We developed the full digital twin of the 6 DOF UR robot in the virtual reality (VR) environment, allowing the adaptive surface simulation and control of the hybrid display in real-time. The preliminary user study was conducted to evaluate the ability of TeslaMirror to reproduce shape sensations with the under-actuated end-effector. The results revealed that potentially this approach can be used in the virtual systems for rendering versatile VR shapes with high fidelity haptic experience., Comment: Accepted to the ACM SIGGRAPH 2020 conference (Emerging Technologies section), ACM copyright, 2 pages, 3 figures, 1 table

Multimodal interactions provide users with more natural ways to manipulate virtual 3D objects than using traditional input methods. An emerging approach is gaze modulated pointing, which enables users to perform object selection and manipulation in a virtual space conveniently through the use of a combination of gaze and other interaction techniques (e.g., mid-air gestures). As gaze modulated pointing uses different sensors to track and detect user behaviours, its performance relies on the user's perception on the exact spatial mapping between the virtual space and the physical space. An underexplored issue is, when the spatial mapping differs with the user's perception, manipulation errors (e.g., out of boundary errors, proximity errors) may occur. Therefore, in gaze modulated pointing, as gaze can introduce misalignment of the spatial mapping, it may lead to user's misperception of the virtual environment and consequently manipulation errors. This paper provides a clear definition of the problem through a thorough investigation on its causes and specifies the conditions when it occurs, which is further validated in the experiment. It also proposes three methods (Scaling, Magnet and Dual-gaze) to address the problem and examines them using a comparative study which involves 20 participants with 1040 runs. The results show that all three methods improved the manipulation performance with regard to the defined problem where Magnet and Dual-gaze delivered better performance than Scaling. This finding could be used to inform a more robust multimodal interface design supported by both eye tracking and mid-air gesture control without losing efficiency and stability.

This paper presents a low-cost 3D motion capture system using a single camera and passive optical markers. 3D motion capture systems usually demand multiple cameras. Thus, those systems require high-cost hardware devices and large computing power. However, such high-cost systems prevent personal or mobile systems using 3D motion capturing. To overcome this problem, we propose a low-cost motion capture system. The structure of our system is based on a single camera and passive optical markers, which enables the system to be low-cost. As the components of the proposed systems, we introduce 2D optical marker recognition, marker size recognition, and mapping for 3D data extraction. To show the effectiveness and efficiency, we conduct preliminary experiments in terms of accuracy and time-complexity. Experiment results indicate that the proposed system provides 3D motion capturing using a single camera enough to utilize it in 3D interaction.

Today there is a high variety of haptic devices capable of providing tactile feedback. Although most of existing designs are aimed at realistic simulation of the surface properties, their capabilities are limited in attempts of displaying shape and position of virtual objects. This paper suggests a new concept of distributed haptic display for realistic interaction with virtual object of complex shape by a collaborative robot with shape display end-effector. MirrorShape renders the 3D object in virtual reality (VR) system by contacting the user hands with the robot end-effector at the calculated point in real-time. Our proposed system makes it possible to synchronously merge the position of contact point in VR and end-effector in real world. This feature provides presentation of different shapes, and at the same time expands the working area comparing to desktop solutions. The preliminary user study revealed that MirrorShape was effective at reducing positional error in VR interactions. Potentially this approach can be used in the virtual systems for rendering versatile VR objects with wide range of sizes with high fidelity large-scaleshape experience., Accepted to the 25th ACM Symposium on Virtual Reality Software and Technology (VRST'19), ACM copyright

Virtual museum is a place where users can explore museum collection freely. In this study, we are discussing the 3D interactions presented in a virtual museum application using hand-sensing sensor named Leap Motion. In making 3D interaction, some hand gestures are needed to functions any interact in virtual world. To prevent miss-occurring in 3D interaction, it is necessary to do a hand pattern classification to improve accuracy and make it more precision so as not to reduce the quality of immersion in the virtual world. The classification method used in this study is K-Nearest Neighbor (KNN) classification methods. KNN is a method that is quite popular and simple. The first step is data acquisition processing that is used as training data using Leap Motion Controller which takes vector value data (x, y, z) from each fingertip. Then the data normalization process is carried out to facilitate the next process which is feature extraction process. Features are being extracted including angle value between fingers, angle value between fingertips, angle between fingertips and palms, distance vector between fingertips and palms, and elevation value between fingertips and palms. After that, extracted data are being trained and classified using K-Nearest Neighbor (KNN).

A 3D user interface can be adapted in multiple ways according to each user’s needs, skills and preferences. Such adaptation can consist in changing the user interface layout or its interaction techniques. Personalization systems which are based on user models can automatically determine the configuration of a 3D user interface in order to fit a particular user. In this paper, we propose to explore the use of machine learning in order to propose a 3D selection interaction technique adapted to a target user. To do so, we built a dataset with 51 users on a simple selection application in which we recorded each user profile, his/her results to a 2D Fitts Law based pre-test and his/her preferences and performances on this application for three different interaction techniques. Our machine learning algorithm based on Support Vector Machines (SVMs) trained on this dataset proposes the most adapted interaction technique according to the user profile or his/her result to the 2D selection pre-test. Our results suggest the interest of our approach for personalizing a 3D user interface according to the target user but it would require a larger dataset in order to increase the confidence about the proposed adaptations.

Eye gaze involves the coordination of eye and head movement to acquire gaze targets, but existing approaches to gaze pointing are based on eye-tracking in abstraction from head motion. We propose to leverage the synergetic movement of eye and head, and identify design principles for Eye&Head gaze interaction. We introduce three novel techniques that build on the distinction of head-supported versus eyes-only gaze, to enable dynamic coupling of gaze and pointer, hover interaction, visual exploration around pre-selections, and iterative and fast confirmation of targets. We demonstrate Eye&Head interaction on applications in virtual reality, and evaluate our techniques against baselines in pointing and confirmation studies. Our results show that Eye&Head techniques enable novel gaze behaviours that provide users with more control and flexibility in fast gaze pointing and selection.

Objective:The study examines the factors determining the movement time (MT) of positioning an object in an immersive 3D virtual environment.Background:Positioning an object into a prescribed area is a fundamental operation in a 3D space. Although Fitts’s law models the pointing task very well, it does not apply to a positioning task in an immersive 3D virtual environment since it does not consider the effect of object size in the positioning task.Method:Participants were asked to position a ball-shaped object into a spherical area in a virtual space using a handheld or head-tracking controller in the ray-casting technique. We varied object size (OS), movement amplitude (A), and target tolerance (TT). MT was recorded and analyzed in three phases: acceleration, deceleration, and correction.Results:In the acceleration phase, MT was inversely related to object size and positively proportional to movement amplitude. In the deceleration phase, MT was primarily determined by movement amplitude. In the correction phase, MT was affected by all three factors. We observed similar results whether participants used a handheld controller or head-tracking controller. We thus propose a three-phase model with different formulae at each phase. This model fit participants’ performance very well.Conclusion:A three-phase model can successfully predict MT in the positioning task in an immersive 3D virtual environment in the acceleration, deceleration, and correction phases, separately.Application:Our model provides a quantitative framework for researchers and designers to design and evaluate 3D interfaces for the positioning task in a virtual space.

Recently, a new form of telexistence is achieved by recording images with cameras on an unmanned aerial vehicle (UAV) and displaying them to the user via a head mounted display (HMD). A key problem here is how to provide a free and natural mechanism for the user to control the viewpoint and watch a scene. To this end, we propose an improved rate-control method with an adaptive origin update (AOU) scheme. Without the aid of any auxiliary equipment, our scheme handles the self-centering problem. In addition, we present a full 6-DOF viewpoint control method to manipulate the motion of a stereo camera, and we build a real prototype to realize this by utilizing a pan-tilt-zoom (PTZ) which not only provides 2-DOF to the camera but also compensates the jittering motion of the UAV to record more stable image streams.

We present design and impact studies of visual feedback for virtual grasping. The studies suggest new or updated guidelines for feedback. Recent grasping techniques incorporate visual cues to help resolve undesirable visual or performance artifacts encountered after real fingers enter a virtual object. Prior guidelines about such visuals are based largely on other interaction types and provide inconsistent and potentially-misleading information when applied to grasping. We address this with a two-stage study. In the first stage, users adjusted parameters of various feedback types, including some novel aspects, to identify promising settings and to give insight into preferences regarding the parameters. In the next stage, the tuned feedback techniques were evaluated in terms of objective performance (finger penetration, release time, and precision) and subjective rankings (visual quality, perceived behavior impact, and overall preference). Additionally, subjects commented on the techniques while reviewing them in a final session. Performancewise, the most promising techniques directly reveal penetrating hand configuration in some way. Subjectively, subjects appreciated visual cues about interpenetration or grasp force, and color changes are most promising. The results enable selection of the best cues based on understanding the relevant tradeoffs and reasonable parameter values. The results also provide a needed basis for more focused studies of specific visual cues and for choosing conditions in comparisons to other feedback modes, such as haptic, audio, or multimodal. Considering results, we propose that 3D interaction guidelines must be updated to capture the importance of interpenetration cues, possible performance benefits of direct representations, and tradeoffs involved in cue selection.

Computers have become ubiquitous and integrated into our day-to-day activities. Researchers have been exploring mechanisms for interacting with computers using natural means in natural environments. Water interaction is a perfect example. This paper presents our attempt to use foot gestures performed in water as an interaction mechanism. It is an extension of our previous study for detecting objects in a water vessel. An experiment was performed to determine which foot-based gestures are suitable for implementation, and we proceeded to recognize a selected set of gestures using machine-learning techniques. We present our findings regarding which algorithms provide the best recognition rates.

International audience; We present a family of three interactive Context-Aware Selection Techniques (CAST) for the analysis of large 3D particle datasets. For these datasets, spatial selection is an essential prerequisite to many other analysis tasks. Traditionally, such interactive target selection has been particularly challenging when the data subsets of interest were implicitly defined in the form of complicated structures of thousands of particles. Our new techniques SpaceCast, TraceCast, and PointCast improve usability and speed of spatial selection in point clouds through novel context-aware algorithms. They are able to infer a user's subtle selection intention from gestural input, can deal with complex situations such as partially occluded point clusters or multiple cluster layers, and can all be fine-tuned after the selection interaction has been completed. Together, they provide an effective and efficient tool set for the fast exploratory analysis of large datasets. In addition to presenting Cast, we report on a formal user study that compares our new techniques not only to each other but also to existing state-of-the-art selection methods. Our results show that Cast family members are virtually always faster than existing methods without tradeoffs in accuracy. In addition, qualitative feedback shows that PointCast and TraceCast were strongly favored by our participants for intuitiveness and efficiency.

Virtual simulation and 3D interaction have shown great potentials in a variety of domains for our future life. For a virtual fish swarm simulation system, the simulation of cohesion behaviors of fish swarm and the interaction between human and fish swarm are two key components to create immersive interactive experiences. However, it is a huge challenge to create a realistic fish swarm simulation system while providing a natural and comfortable interaction. In this paper, we propose a method for immersive virtual fish swarm simulation based on infrared sensors. Based on dynamic weight constraints, we propose a particle swarm optimization method for fish swarm cohesion simulation, which separates a particle swarm by the state of each particle and dynamically controls the particle swarm, making the movement behavior of virtual fish more realistic. In addition, an interactive fast skinning method is proposed for cartoon fishes, which leverages image segmentation, Optical Character Recognition (OCR) and bone skinning are used to generate cartoon fishes based on user-created colors. With infrared sensors, we propose a method for virtual fish swarm interaction, where the positions of human skeleton are processed by an action analyzer, achieving real-time user interactions with fish swarms. With all the proposed techniques integrated in a system, the experimental results show that our method is feasible and effective.

Physical walking is consistently considered a natural and intuitive way to acquire viewpoints in a virtual environment. However, research findings also show that walking requires cognitive resources. To understand how this tradeoff affects the interaction design for virtual environments; we evaluated the performance of 32 participants, ranging from 18 to 44 years old, in a demanding visual and spatial task. Participants wearing a virtual reality (VR) headset counted features in a complex 3D structure while walking or while using a 3D interaction technique for manipulation. Our results indicate that the relative performance of the interfaces depends on the spatial ability and game experience of the participants. Participants with previous game experience but low spatial ability performed better using the manipulation technique. However, walking enabled higher performance for participants with low spatial ability and without significant game experience. These findings suggest that the optimal design choices for demanding visual tasks in VR should consider both controller experience and the spatial ability of the target users.

In this paper, we propose a 3D interactive method which can track handheld rigid object without markers and recognize human pose simultaneously. Based on depth data of Kinect v2.0, an improved bilateral filtering utilized to solve the failure of recognition caused by occlusion, then the target point cloud can be obtained after segment and tracked by Interactive Closet Point (ICP) algorithm. Through this method, users can operate the actual rigid object with both hands, change posture to implement multiple interactions with 3D scenes. Thus, the method breaks the limitations of a fixed interactive medium such as joystick and is suitable to promote to general application.

Measuring hand and finger movements, and interaction forces, is important for the assessment of tasks in daily life. This thesis proposes a new on-body assessment system that allows the measurement of movements and interaction forces of the hand, fingers and thumb. The first thesis objective concerns the development, evaluation and validation of an inertial and magnetic sensing system for the measurement of hand and finger kinematics. The second objective concerns the assessment of the dynamic interaction between human hand and environment using combined force and movement sensing. Chapter 2 describes the inertial and magnetic sensing hardware, and kinematic estimation algorithms for a sensing system which can be attached to the hand, fingers and the thumb. Chapter 3 reports an extensive comparison of this inertial sensing system against a passive opto-electronic marker system under for different tasks that mimic situations in activities of daily living. Chapter 4 describes a new method to ease the typical anatomical segment and sensor calibration procedures by estimating these parameters implicitly along with the estimation of the state variables. In addition, the method incorporates information form chained linkages to circumvent the usage of magnetometers for heading estimation. Chapter 5 presents a solution to estimate the full pose (3D position and 3D orientation) of the hand with respect to the sternum of the body using inertial sensors, magnetometers and a permanent magnet. Contrary to the previous chapters is the magnetic information used to yield a drift free position estimate. Chapter 6 concerns the second research objective which is about the assessment of the physical interaction between the human hand and environmental objects. Dedicated sensors have been applied to measure 3D interaction forces. This hardware has been combined with the inertial hardware presented in the previous chapters and attached to the finger and thumb tips to measure interaction forces and finger motions simultaneously. Eventually the system is used to quantify object dynamics using the information obtained during manipulation.

National audience; Tasks and contexts of interaction within virtual environments are more and more varied and complex, and sometimes unpredictable. The design of static 3D user interaction techniques is not always adapted. Our researches aim at providing "context-aware" 3DUI. Having analyzed several solutions and knowledge representation and reasoning methods, we conceived and developed a generic decision engine. Knowledge representation is based on an ontology and Conceptual Graphs. Semantic reasoning uses 1st order logic. It can communicate with existing applications through a set of software tools: the information of context is gathered by sensors, whereas the multimodal assistance is realized by effectors. This process was applied for the adaptive assistance to the selection of objects, and for the offline analysis of activity.; Les tâches et les contextes d’interaction dans les environnements virtuels sont de plus en plus variées et complexes, et parfois imprévisibles. Les techniques d’interactions 3D figées lors de la conception n’étant pas toujours adaptées, nos recherches visent une assistance aux interactions 3D “sensible au contexte” ou “adaptative”. Après avoir analysé différentes solutions et méthodes de représentation et de raisonnement, nous avons conçu et développé un moteur de décision générique qui utilise une ontologie et des graphes conceptuels pour représenter les connaissances, et la logique du 1er ordre pour conduire un raisonnement sémantique. Il peut communiquer avec des applications de réalité virtuelle existantes à travers des outils logiciels : les informations de contexte sont rassemblées par des capteurs, tandis que l’assistance multimodale est réalisée par des effecteurs. Ce processus a été appliqué pour l’assistance adaptative à la sélection d’objets, et pour l’analyse d’activité hors-ligne.

The implementation of three-dimensional visual languages requires a wide range of conceptual and technical knowledge on issues for 3D graphics and textual language processing. Our generator framework DEViL3D incorporates such knowledge and supports the design of visual 3D languages and their implementation from high-level specifications. Such 3D languages arise from different modeling domains that make use of three-dimensional representations, e.g. the "ball-and-stick" models of molecules. The front-end of a 3D language implementation is a dedicated 3D graphical structure editor, which offers interaction and navigation techniques to construct programs in their domain. These techniques allow to manipulate the 3D program directly using operations to insert, move, and restructure objects. We have developed canned solutions for all such techniques that are encapsulated in visual patterns, which are provided by our generator. The designer of a particular 3D language only has to apply visual patterns to constructs of the abstract syntax, which defines the basic structure of the language. We have complemented our development with a usability study. Participants had to solve several tasks with different interaction or navigation techniques. Furthermore, we equipped the 3D editors with the opportunity to let users gain an immersive 3D perception by using stereoscopic hardware.

International audience; In mobile applications, it is crucial to provide intuitive means for 2D and 3D interaction. A large number of techniques exist to support a natural user interface (NUI) by detecting the user’s hand posture in RGB+D (depth) data. Depending on the given interaction scenario and its environmental properties, each technique has its advantages and disadvantages regarding accuracy and the robustness of posture detection. While the interaction environment in a desktop setup can be constrained to meet certain requirements, a handheld scenario has to deal with varying environmental conditions. To evaluate the performance of techniques on a mobile device, a powerful software framework was developed that is capable of processing and fusing RGB and depth data directly on a handheld device. Using this framework, five existing hand posture recognition techniques were integrated and systematically evaluated by comparing their accuracy under varying illumination and background. Overall results reveal best recognition rate of posture detection for combined RGB+D data at the expense of update rate. To support users in choosing the appropriate technique for their specific mobile interaction task, we derived guidelines based on our study. In the last step, an experimental study was conducted using the detected hand postures to perform the canonical 3D interaction tasks selection and positioning in a mixed reality handheld setup.

We present a novel signaling method for head-mounted displays, which surpasses eye (pupil) and delivers guiding light signals directly to retina through tissue near the eyes. This method preserves full visual acuity on the display and does not block view to the scene, while also delivering additional visual signals.

International audience; 3D object selection and manipulation is one of the essential features for any augmented reality (AR) system. However, distant object selection and manipulation still suffer from lack of accuracy and precision. This paper introduces an alternate 3D interaction technique for selection and manipulation distant 3D object in in immersive video see-through AR. The proposed interaction technique offers a high precision when selecting and manipulation distant objects thanks to the zooming-based idea. This later, allows bringing closer both real and virtual objects during maintaining the spatio-temporal registration between the virtual and the real scenes. The evaluation of our proposed approach and the comparison with other well-known techniques are given at the end of this paper.

We present a low-cost 3D tracking system for virtual reality, gesture modeling, and robot manipulation applications which require fast and precise localization of headsets, data gloves, props, or controllers. Our system removes the need for cameras or projectors for sensing, and instead uses cheap LEDs and printed masks for illumination, and low-cost photosensitive markers. The illumination device transmits a spatiotemporal pattern as a series of binary Gray-code patterns. Multiple illumination devices can be combined to localize each marker in 3D at high speed (333Hz). Our method has strengths in accuracy, speed, cost, ambient performance, large working space (1m-5m) and robustness to noise compared with conventional techniques. We compare with a state-of-the-art instrumented glove and vision-based systems to demonstrate the accuracy, scalability, and robustness of our approach. We propose a fast and accurate method for hand gesture modeling using an inverse kinematics approach with the six photosensitive markers. We additionally propose a passive markers system and demonstrate various interaction scenarios as practical applications.

In this paper, we propose a system that realizes remote control of a computer with small hand gestures. Distant pointing is realized using a 3D hand pointing recognition algorithm that obtains position and direction of the pointing hand. We show the effectiveness of the system by constructing three types of user interfaces considering the accuracy of distant pointing in the current system. We created a tile layout interface for rough selection operation, a pie menu interface for detailed operation, and a viewer interface for document browsing.

We present a re-imaged swept volume display enabling mid-air interaction with 3D floating objects without requiring a head-mounted apparatus. The presence of the volume is so strong that everyone reaches out to touch it, at which point the illusion is broken. To resolve this, we implemented interaction techniques to mitigate occlusion conflicts between the hand and virtual volume during direct manipulation. Our long term goal is to prototype direct spatial 3D manipulation techniques that highlight direct manipulation scenarios such as viewing 3D scans, previewing 3d prints, and product design visualization.

We created a system that enables a user to touch and interact with 3D images formed by a light source device in mid-air. A user can get the haptic feedback without wearing a special device by using focused airborne ultrasound. The pattern of 3D images of 36864 dots and tactile feedback points are fully programmable and updated at the refresh-rate of 18 Hz. Using this system, we demonstrate the feasibility of handling a 3D virtual object.

In the previous study, a new interactive system called HaptoClone was proposed. In HaptoClone system, a user can interact with optically copied objects from the adjacent workspace with haptic feedback. In this study, we propose an improved version of the HaptoClone system called HaptoCloneAR, which superimposes a virtual 2D screen on the copied objects by using half-mirrors. The system can display 2 different images independently to each workspace. In this paper, we show a basic configuration of the HaptoCloneAR system. We demonstrate a feasibility of the proposed system with our prototype.

International audience; Real-time 3D interaction with augmented reality (AR) environments is one of the main features of any AR system. However, selecting and manipulating distant 3D virtual objects in AR still suffer from lack of accuracy and precision. In this paper, we propose an alternate 3D interaction technique called “Zoomin” for selection and manipulation distant objects in immersive video see-through augmented reality. Zoom-in interactiontechnique is based on the idea of zooming the captured images. This allows bringing closer both of real and virtual distant objects, while keeping the spatial registration between the virtual and the real scenes thanks to a robust real-time computer vision algorithm for pose estimation. An evaluation and comparison with other well-known technique are given at the end of this paper, in order to validate our proposed approach.

Many types of virtual reality (VR) systems allow users to use natural, physical head movements to view a 3D environment. In some situations, such as when using systems that lack a fully surrounding display or when opting for convenient low-effort interaction, view control can be enabled through a combination of physical and virtual turns to view the environment, but the reduced realism could potentially interfere with the ability to maintain spatial orientation. One solution to this problem is to amplify head rotations such that smaller physical turns are mapped to larger virtual turns, allowing trainees to view the entire surrounding environment with small head movements. This solution is attractive because it allows semi-natural physical view control rather than requiring complete physical rotations or a fully-surrounding display. However, the effects of amplified head rotations on spatial orientation and many practical tasks are not well understood. In this paper, we present an experiment that evaluates the influence of amplified head rotation on 3D search, spatial orientation, and cybersickness. In the study, we varied the amount of amplification and also varied the type of display used (head-mounted display or surround-screen CAVE) for the VR search task. By evaluating participants first with amplification and then without, we were also able to study training transfer effects. The findings demonstrate the feasibility of using amplified head rotation to view 360 degrees of virtual space, but noticeable problems were identified when using high amplification with a head-mounted display. In addition, participants were able to more easily maintain a sense of spatial orientation when using the CAVE version of the application, which suggests that visibility of the user's body and awareness of the CAVE's physical environment may have contributed to the ability to use the amplification technique while keeping track of orientation.

International audience; Proper integration of sensory cues facilitates 3D user interaction within virtual environments (VEs). Studies showed that the integration of visual and haptic cues follows maximum likelihood estimation (MLE). Little effort focuses however on the mechanism of integrating force and vibrotactile cues. We thus investigated MLE's suitability for integrating these cues. Within a VE, human users undertook 3D interaction of navigating a flying drone along a high-voltage transmission line for inspection. The users received individual force or vibrotactile cues, and their combinations in collocated and dislocated settings. The users' task performance including completion time and accuracy was assessed under each individual cue and setting. The presence of the vibrotactile cue promoted a better performance than the force cue alone. This agreed with the applicability of tactile cues for sensing 3D surfaces, herein setting a baseline for using MLE. The task performance under the collocated setting indicated a degree of combining the individual cues. In contrast, the performance under the dislocated setting was alike under the individual vibrotactile cue. These observations imply a possible role of MLE in integrating force and vibrotactile cues for 3D user interaction within VEs.

The target picking in 3D volumetric data is a common interaction problem in a variety of domains such as medicine, engineering, and physics. When a user clicks on a 2D point on the screen rendering, the ray is cast through the volume at an angle perpendicular to the screen. The problem lies in determining the intended location of picking, that is the end point of the ray. We introduce picking for 3D volumetric data by utilizing pressure and angle of the digital stylus pen on a touchscreen device. We map both pressure and angle of the stylus to the depth of the selector widget to help target acquisition. We evaluate several methods of selection: single finger selection, stylus with pressure but without an angle, and stylus with both angle and pressure. We decouple rotation of the volumetric data, and control for two target sizes. We report significant benefits of digital stylus picking methods and a relation between the methods and target sizes.

We describe a mid-air, watch-based 3D interaction technique called Watchcasting. The technique enables target selection and translation by mapping z-coordinate position to forearm rotation. By replicating a large display 3D selection study, we show that Watchcasting provides comparable performance to smartphone (Smartcasting) and electrical impedance myography (Myopoint) techniques. Our work demonstrates that an off-the-shelf smartwatch is a practical alternative for 3D interaction using specialized devices.

The use of natural interfaces improves significantly aspects related to human-computer interaction and consequently the productivity and the overall performance. In this paper we present a novel framework to interact with data elements presented in a 3D space. The system provides two mechanisms to interact using 2D and 3D gestures based on data provided by Kinect and on hand detection and gesture interpretation algorithms. The proposed architecture is analysed indicating that 3D interaction with information is possible, and provides advantages over a 2D interaction over the same problem. Finally, two sets of experiments were performed to evaluate 2D and 3D interaction styles based on natural interfaces focusing on traditional interaction with 3D databases.

In this paper, we introduce a novel, real-time and robust hand tracking system, capable of tracking the articulated hand motion in full degrees of freedom (DOF) using a single depth camera. Unlike most previous systems, our system is able to initialize and recover from tracking loss automatically. This is achieved through an efficient two-stage k-nearest neighbor database searching method proposed in the paper. It is effective for searching from a pre-rendered database of small hand depth images, designed to provide good initial guesses for model based tracking. We also propose a robust objective function, and improve the Particle Swarm Optimization algorithm with a resampling based strategy in model based tracking. It provides continuous solutions in full DOF hand motion space more efficiently than previous methods. Our system runs at 40 fps on a GeForce GTX 580 GPU and experimental results show that the system outperforms the state-of-the-art model based hand tracking systems in terms of both speed and accuracy. The work result is of significance to various applications in the field of human---computer-interaction and virtual reality.

In this paper, we propose a novel OLED display system by adding depth-sensing capability to the display system. Coded apertures formed in OLED pixels with transmissive windows encode the radiation from a scene and the scene is reconstructed by an appropriate decoding pattern. Further, depth information is estimated from multiple coded images. In order to prove the feasibility of the proposed system, a 19” imaging system with transmissive windows within pixels was constructed and tested. Experimental results confirm that the proposed system can reconstruct the scene and accurately estimate the depth of an object in front of the display system.

We propose a 3D interaction and autostereoscopic display system that use gesture recognition, which can manipulate virtual objects in the scene directly by hand gestures and can display objects in 3D stereoscopy. The system consists of a gesture recognition and manipulation part as well as an autostereoscopic display as an interactive display part. To manipulate the 3D virtual scene, a gesture recognition algorithm is proposed, which use spatial-temporal sequences of feature vectors to match predefined gestures. To get smooth 3D visualization, we utilize the programmable graphics pipeline in graphic processing unit to accelerate data processing. We develop a prototype system for 3D virtual exhibition. The prototype system reaches frame rates of 60 fps and operates efficiently with a mean recognition accuracy of 90%.

The use of 3D computer graphics is important in a very wide range of applications. However, user interaction with 3D applications is still challenging and often does not lend itself to established techniques that have been developed primarily for 2D desktop interaction. Meanwhile, 3D user interfaces that rely on tracking hand-held devices or fiducial markers attached to the user are cumbersome or entirely inappropriate in some situations. These challenges may be addressed by refining and building on the increasing use of freehand gestural input, i.e. without markers or hand-held devices, to extend the fluidity and immediacy of today's 2D touch-based interactions. In this paper, we analyze the characteristics of freehand gestural 3D interaction, and report a set of 3 related evaluation studies focused on the fundamental user interface task of object selection. We found that interaction design requiring a high accuracy single action are not appropriate for freehand gestural selection, while separating it into several connected low demand operations could be a potential solution; that our Reach technique is accurate and potentially useful for option selection tasks with freehand gesture; and that strong directional effects influence performance and usability of both 2D and 3D option selection. We propose guidelines for designers of 3D freehand gestural interaction based on our evaluation results.

This paper presents an augmented reality (AR) based rehabilitation exercise system to enhance the motor function of the hands for the paretic/hemi-paretic patient. The existing rehabilitation systems rely on mechanical apparatus for palsy rehabilitation, but we aim to use the rehabilitation system at home with easy configuration and minimized equipment by the computer vision based approach. The proposed method evaluates the interaction status of the fingertip action by using the position and the contact of the fingertip markers. We obtain the 2D positions of the fingertip markers from a single camera, and then transform the 3D positions from the calibrated camera space by using an ARToolKit marker. We adopt simple geometric calculation by the conversion of the 2D interest points into the 3D interaction points for the simple interactive task in AR environment. Some experimental results show that the proposed method is practical and simply applicable to the applications with personal AR interaction.

The generation of digital information is continuing at an accelerated pace, and the exploration of relationships within such enormous volumes of data is becoming increasingly difficult. Information visualization methods directly address the requirements of human perception to help users analyze complex relationships, and graphical hierarchy trees are employed to present the results. Conventional methods only provide a fixed degree of detail, despite the requirements of users to alter the manner in which data is viewed. Magic Eye View, a well-known information visualization tool, incorporates a three-dimensional interactive function allowing users to control the degree of detail with which they view data. Unfortunately, this system fails to provide a number of crucial focus+context features. To address these shortcomings, we propose a novel 3D information visualization method that enables users to control the focus point three-dimensionally and view the information at a preferred degree of detail. It also provides crucial focus+context features that help users to comprehend the data. Simulation results have shown significant advantages over existing approaches through the incorporation of these crucial features.

We describe our approach that derives reliable 6-DOF information including the translation and the rotation of a rigid marker in a 3D space from a set of insufficient 5-DOF measurements. As a practical example, we carefully constructed a prototype and its design and evaluated it in our 3D dexterous motion-tracking system, IM6D, which is our novel real-time magnetic 3D motion-tracking system that uses multiple identifiable, tiny, lightweight, wireless, and occlusion-free markers. The system contains two key technologies; a 6-DOF computation algorithm and a marker design for 6D marker. The 6-DOF computation algorithm computes the result of complete 6-DOF information including translation and rotation in 3D space for a single rigid marker that consists of three LC coils. We propose several possible approaches for implementation, including geometric, matrix-based kinematics, and computational approaches. In addition, we introduce workflow to find an optimal marker design for the system to achieve the best compromise between its smallness and accuracy based on the tracking principle. We experimentally compare the performances of some typical marker prototypes with different layouts of LC coils. Finally, we also show another experimental result to prove the effectiveness of the results from the solutions in these two problems.

Accurate fingertip detection is important for hand-based human computer interaction. Different from prior methods based on depth image, in this paper we propose a novel scheme for accurate fingertip detection from binocular mask images without explicitly computing the depth map. To demonstrate our proposed scheme, we build a new hand dataset containing synthetic and real binocular images. A deep convolutional neural network (CNN) is utilized as a baseline method to demonstrate the proposed scheme. The mask images are extracted from binocular images and fed into the CNN to predict the 3D positions of fingertips and palm center. Experiments show that this method achieves the mean error of 8.30mm on synthetic data and 4.64mm on real data, which is promosing for accurate 3D interaction. The proposed scheme runs at 130fps on a CPU and is promising for real-time applications.