Search

Interaction is one of the important topics to be discussed since it includes the interface where the end-user communicates with the augmented reality (AR) system. In handheld AR interface, the traditional interaction techniques are not suitable for some AR applications due to the different attributes of handheld devices that always refer to smartphones and tablets. Currently interaction techniques in handheld AR are known as touch-based technique, mid-air gesture-based technique and device-based technique that can led to a wide discussion in related research areas. However, this paper will focus to discover the device-based interaction technique because it has proven in the previous studies to be more suitable and robust in several aspects. A novel device-based 3D object rotation technique is proposed to solve the current problem in performing 3DOF rotation of 3D object. The goal is to produce a precise and faster 3D object rotation. Therefore, the determination of the rotation amplitudes per second is required before the fully implementation. This paper discusses the implementation in depth and provides a guideline for those who works in related to device-based interaction.

Low-cost virtual-reality (VR) head-mounted displays (HMDs) with the integration of smartphones have brought the immersive VR to the masses, and increased the ubiquity of VR. However, these systems are often limited by their poor interactivity. In this paper, we present GestOnHMD, a gesture-based interaction technique and a gesture-classification pipeline that leverages the stereo microphones in a commodity smartphone to detect the tapping and the scratching gestures on the front, the left, and the right surfaces on a mobile VR headset. Taking the Google Cardboard as our focused headset, we first conducted a gesture-elicitation study to generate 150 user-defined gestures with 50 on each surface. We then selected 15, 9, and 9 gestures for the front, the left, and the right surfaces respectively based on user preferences and signal detectability. We constructed a data set containing the acoustic signals of 18 users performing these on-surface gestures, and trained the deep-learning classification pipeline for gesture detection and recognition. Lastly, with the real-time demonstration of GestOnHMD, we conducted a series of online participatory-design sessions to collect a set of user-defined gesture-referent mappings that could potentially benefit from GestOnHMD.

As smartphone screens have grown in size, single-handed use has become more cumbersome. Interactive targets that are easily seen can be hard to reach, particularly notifications and upper menu bar items. Users must either adjust their grip to reach distant targets, or use their other hand. In this research, we show how gaze estimation using a phone’s user-facing camera can be paired with IMU-tracked motion gestures to enable a new, intuitive, and rapid interaction technique on handheld phones. We describe our proof-of-concept implementation and gesture set, built on state-of-the-art techniques and capable of self-contained execution on a smartphone. In our user study, we found a mean euclidean gaze error of 1.7 cm and a seven-class motion gesture classification accuracy of 97.3%.

Gesture-based interfaces offer a suitable platform for interactions in Virtual Environments (VE). However, the difficulties involved in learning and making of distinct gestures affect the performance of an interactive system. By incorporating computer vision in Virtual Reality (VR), this paper presents an intuitive interaction technique where the states and positions of eyes are traced for interaction. With comparatively low cognitive load, the technique offers an easy to use interaction solution for VR applications. Unlike other gestural interfaces, interactions are performed in distinct phases where transition from one phase to another is enacted with simple blink of eyes. In an attained phase, interaction along an arbitrary axis is performed by a perceptive gesture of head; rolling, pitching or yawing. To follow the trajectory of eyes in real time, coordinates mapping is performed dynamically. The proposed technique is implemented in a case-study project; EBI (Eyes Blinking based Interaction). In the EBI project, real time detection and tracking of eyes are performed at the back-end. At the front-end, virtual scene is rendered accordingly by using the OpenGL library. To assess accuracy, usability and cognitive load of the proposed technique, the EBI project is evaluated 280 times in three different evaluation sessions. With an ordinary camera, an average accuracy of 81.4% is achieved. However, assessment made by using a high-quality camera revealed that accuracy of the system could be raised to a higher level. As a whole, findings of the evaluations support applicability of the technique in the emerging domains of VR.

Speech input, such as voice assistant and voice message, is an attractive interaction option for mobile users today. However, despite its popularity, there is a use limitation for smartphone speech input: users need to press a button or say a wake word to activate it before use, which is not very convenient. To address it, we match the motion that brings the phone to mouth with the user's intention to use voice input. In this paper, we present ProxiTalk, an interaction technique that allows users to enable smartphone speech input by simply moving it close to their mouths. We study how users use ProxiTalk and systematically investigate the recognition abilities of various data sources (e.g., using a front camera to detect facial features, using two microphones to estimate the distance between phone and mouth). Results show that it is feasible to utilize the smartphone's built-in sensors and instruments to detect ProxiTalk use and classify gestures. An evaluation study shows that users can quickly acquire ProxiTalk and are willing to use it. In conclusion, our work provides the empirical support that ProxiTalk is a practical and promising option to enable smartphone speech input, which coexists with current trigger mechanisms.

As ubiquitous computing provide up-graded smart environments where humans desire to create various types of interaction for many kinds of media and information, the research in the area of Human-Computer Interaction (HCI) is being emphasized to satisfy a more convenient user interface. In particular, the gesture interaction technique has been one of the important research areas under ubiquitous computing environment since it can only utilize widespread consumer video cameras and computer vision techniques without the aid of any other devices to grasp human movements and intentions(Park, 2004; Jung, 2007). Among the gesture interaction techniques, recognition of hand poses and gestures has especially received attention due to great potential to build various and user-centric computer interfaces. The applicability of hand pose recognition is very high in applications where system users can not use existing interface devices such as a keyboard and a mouse since they are required to wear heavy protective gloves for industrial processes. Various types of gesture interfaces have been also presented in three-dimensional games based on virtual reality and these interfaces have enhanced an interest level and creativity within these environments for the users. Humans can distinguish hand poses very quickly through their complex optical systems, while it is very difficult for a computer system to rapidly and accurately understand hand poses. Therefore, many researchers have tried to simulate the human optical system, which can extract objects of interest from complex scenes and understand the context among objects. One of the major factors that disturb automatic gesture recognition is illumination change. The sudden illumination changes lead to the misunderstanding of background and foreground regions. We propose a robust hand recognition technique that can stably extract hand contours even under sudden illumination changes. Figure 1 shows the flowchart for our proposed method. The proposed method acquires the background images for a restricted duration and calculates the mean and standard deviation for the hue and hue-gradient of each pixel within the captured background images. That is, a background model for each pixel is built. The hue and hue-gradient of the input images captured in real-time are calculated and compared to those of the background images. The foreground objects are extracted based on the difference magnitude between those of the input image and the background image. To accurately extract the tight object region of interest, we calculate the eigen value and eigen

Gesture, an important mode of human natural behavior, has the advantages of rich semantics, natural flexibility, simplicity and convenience, and has become a research focus of new human-machine interaction technique. This paper, take the multi-UAV formation human-machine coordinated operation and control as the background, based on the gesture recognition method of surface electromyography(sEMG), and researches the gesture interaction technique for multi-UAV formations, which makes the operators use gestures to complete the flight operation and control of UAV formations without space constraints.

We present HulaMove, a novel interaction technique that leverages the movement of the waist as a new eyes-free and hands-free input method for both the physical world and the virtual world. We first conducted a user study (N=12) to understand users’ ability to control their waist. We found that users could easily discriminate eight shifting directions and two rotating orientations, and quickly confirm actions by returning to the original position (quick return). We developed a design space with eight gestures for waist interaction based on the results and implemented an IMU-based real-time system. Using a hierarchical machine learning model, our system could recognize waist gestures at an accuracy of 97.5%. Finally, we conducted a second user study (N=12) for usability testing in both real-world scenarios and virtual reality settings. Our usability study indicated that HulaMove significantly reduced interaction time by 41.8% compared to a touch screen method, and greatly improved users’ sense of presence in the virtual world. This novel technique provides an additional input method when users’ eyes or hands are busy, accelerates users’ daily operations, and augments their immersive experience in the virtual world.

Surface gestures have been the dominant form of interaction for mobile devices with touchscreen. From our survey of current consumer mobile Augmented Reality (AR) applications, we found that these applications also adopted the surface gestures metaphor as their interaction technique. However, we believe that designers should be able to utilize the affordance of the three-dimensional interaction space that AR provides. We compared two interaction techniques of surface and motion gestures in a Pokemon GO-liked mobile AR game that we developed. Ten participants experienced both techniques to capture three sizes of Pokemon. The comparison examined the two types of gestures in terms of accuracy, game experience, and subjective ratings on goodness, ease of use, and engagement. It was found that the motion gesture provided better engagement and game experience, while the surface gesture was more accurate and easier to use.

We present Headbang, an interaction technique that enriches touch input on handheld devices through slight head movement gestures. This way, users can easily execute shortcuts, like Copy, Paste, or Share, to on-screen targets while touching them. Headbang utilizes the capabilities of commodity smartphones to track the user’s head with their front facing cameras. We evaluated Headbang in two studies and show that the system can be reliably used while sitting and walking and offers a similar accuracy and speed as touch interaction.

Recently, a new breed of mobile virtual reality (dubbed as &ldquo, EasyVR&rdquo, in this work), has appeared in the form of conveniently clipping on a non-isolating magnifying lenses on the smartphone, still offering a reasonable level of immersion to using the isolated headset. Furthermore, such a form factor allows the fingers to touch the screen and select objects quite accurately, despite the finger(s) being seen unfocused over the lenses. Many navigation techniques have existed for both casual smartphone 3D applications using the touchscreen and immersive VR environments using the various controllers/sensors. However, no research has focused on the proper navigation interaction technique for a platform like EasyVR which necessitates the use of the touchscreen while holding the display device to the head and looking through the magnifying lenses. To design and propose the most fitting navigation method(s) with EasyVR, we mixed and matched the conventional touchscreen based and headset oriented navigation methods to come up with six viable navigation techniques&mdash, more specifically for selecting the travel direction and invoking the movement itself&mdash, including the use of head-rotation, on-screen keypads/buttons, one-touch teleport, drag-to-target, and finger gestures. These methods were experimentally compared for their basic usability and the level of immersion in navigating in 3D space with six degrees of freedom. The results provide a valuable guideline for designing/choosing the proper navigation method under different navigational needs of the given VR application.

In this paper, we present a new controller-based interaction technique on the Microsoft HoloLens to support communication for the early phase satellite design at the Concurrent Engineering Facility (CEF). We design a virtual satellite with virtual moveable objects utilizing two different interaction methods: the default hand gesture-based interaction method and a novel controller-based interaction method for rotation and translation of satellite components in immersive augmented reality. In order to evaluate our method, we conduct a perceptual study with 12 participants. We apply multiple performance metrics for each user on both methods. Additionally, we measure user preferences and ease of use. Our results show that our controller-based method is significantly more precise for placing objects (consisting of position and orientation). Furthermore, it is less time consuming than the hand gesture-based method and more preferred by the participants.

Unlike large and dangerous industrial robots at production lines in factories that are strictly fenced off, collaborative robots are smaller and safer and can be installed adjacent to human workers and collaborate with them. However, controlling and teaching new moves to collaborative robots can be difficult and time-consuming when using existing methods, such as pressing buttons on a teaching pendant and physically grabbing and moving the robot via direct teaching. We present Robot Telekinesis, a novel robot interaction technique that lets the user remotely control the movement of the end effector of a robot arm with unimanual and bimanual hand gestures that closely resemble handling a physical object. Through formal evaluation, we show that using a teaching pendant is slow and confusing and that direct teaching is fast and intuitive but physically demanding. Robot Telekinesis is as fast and intuitive as direct teaching without the need for physical contact or physical effort.

Tongue-computer interaction techniques create a new pathway between human mind and computer, with particular utility for people with upper limb impairment. The high dexterity and resilience of the tongue make it a good candidate for interacting with computers. This paper introduces a new interaction technique, camera-based tongue computer interface (CBTCI), which employs tongue without any direct physical contact required. Through a two-phase study, the CBTCI was evaluated and its interaction problems were identified and discussed. In the first phase, the performance of the CBTCI prototype was evaluated through two user tests. The participants’ behaviors were observed throughout the first phase and analyzed to scaffold the study of the design problems in gesture-based tongue computer interaction. The Phase 2 study investigated the usability problems of CBTCI which were reflected through the user behavior and participants’ feedback; specifically the exploration of referential techniques to make users aware of their tongue position and adjust their gesture. Pros and cons of the referential strategies are discussed to foster future assistive tongue-computer interface design.

Handheld mobile devices referring to mobile phones and tablets become the major output medium for augmented reality (AR), which have seen significant growth in popularity and usage among the public due to the growing release of consumer-oriented communication products nowadays, especially touchscreen smartphones. Unlike traditional desktop or tabletop AR (large display-based) and head-mounted display-based AR systems, small display-based AR (handheld mobile display) requires different interaction techniques that mostly utilize single-hand interaction as well as the limitation of small screen display and limited activity time due to the battery operation hour of handheld mobile devices. However, in handheld mobile AR, research is still lacking, especially research that focuses on 3D interaction in handheld mobile AR for virtual object manipulation. Thus, this paper provides an overview of 3D interaction techniques in handheld mobile AR with critical analysis. First, we describe three main interaction technique categories that applicable in handheld mobile AR, which is touch-based interaction, mid-air gestures-based interaction, and device-based interaction techniques, of their basic concepts on 3D object manipulation. Then, we classify and systematize the highlighted techniques and discuss the advantages and drawbacks of each. Previous studies for widely used techniques have been studied comprehensively. We then draw up a comparison among the different techniques based on the important elements considered in handheld mobile AR. The aim of this paper is to provide researchers with background information on AR and those who are interested in the field of 3D interaction, realizing each technique category.

User interface has special importance in immersive virtual environments. Interactions based on the simple and conceivable gestures of a hand may enhance immersivity of a Virtual Environment (VE). However, due to the structural issues like small size and complex shape of human hand, recognition of hand gestures are more challenging. This work introduces a novel interaction technique to perform the basic interaction tasks by the simple movement of hand instead of distinct gestures. With an ordinary camera, the fist posture of hand is segmented out from the image stream using the optimal segmentation model. Like pressing a button with a thumb, the status of thumb is traced for the activation or deactivation of the interactions. After the activation of interaction, the trajectory of hand is followed to manipulate a virtual object about an arbitrary axis. Without training and comparison of gestures, the basic interactions required in a VE are performed by the perceptive movement of a hand. By incorporating image processing in the realm of VE, the technique is implemented in a case-study project; FIRST (Feasible Interaction by Recognizing the Status of Thumb). A group of 12 users evaluated the system in a moderate lighting condition. Outcomes of the evaluation revealed that the technique is suitable for Virtual Reality (VR) applications.

We present ScraTouch, an interaction technique using fingernails, as a new input modality for capacitive touch surfaces. We revealed the differences between the fingertip and the nail from the viewpoint of capacitance touch sensing. Differentiating between fingertip and fingernail touches requires only tens of milliseconds worth of shunt current data from the capacitive touch sensing mechanism, thus external sensors or other hardware are unnecessary. Owing to the small amount of friction generated when a nail touches the surface, not only tapping but also touch gestures by sliding a nail can be easily performed. In our initial investigation, we confirmed that setting a simple threshold on the measured shunt current for recognition works robustly.

Collaborative Virtual Environments (CVEs) offer unique opportunities for human communication. Humans can interact with each other over a distance in any environment and visual embodiment they want. Although deictic gestures are especially important as they can guide other humans' attention, humans make systematic errors when using and interpreting them. Recent work suggests that the interpretation of vertical deictic gestures can be significantly improved by warping the pointing arm. In this paper, we extend previous work by showing that models enable to also improve the interpretation of deictic gestures at targets all around the user. Through a study with 28 participants in a CVE, we analyzed the errors users make when interpreting deictic gestures. We derived a model that rotates the arm of a pointing user's avatar to improve the observing users' accuracy. A second study with 24 participants shows that we can improve observers' accuracy by 22.9%. As our approach is not noticeable for users, it improves their accuracy without requiring them to learn a new interaction technique or distracting from the experience.

Locating places in cities is typically facilitated by handheld mobile devices, which draw the visual attention of the user on the screen of the device instead of the surroundings. In this research, we aim at strengthening the connection between people and their surroundings through enabling mid-air gestural interaction with real-world landmarks and delivering information through audio to retain users' visual attention on the scene. Recent research on gesture-based and haptic techniques for such purposes has mainly considered handheld devices that eventually direct users' attention back to the devices. We contribute a hand-worn, mid-air gestural interaction design with directional vibrotactile guidance for finding points of interest (POIs). Through three design iterations, we address aspects of (1) sensing technologies and the placement of actuators considering users' instinctive postures, (2) the feasibility of finding and fetching information regarding landmarks without visual feedback, and (3) the benefits of such interaction in a tourist application. In a final evaluation, participants located POIs and fetched information by pointing and following directional guidance, thus realising a vision in which they found and experienced real-world landmarks while keeping their visual attention on the scene. The results show that the interaction technique has comparable performance to a visual baseline, enables high mobility, and facilitates keeping visual attention on the surroundings.

Interacting with digital contents in 3D is an essential task in various applications such as modeling packages, gaming, virtual reality, etc. Traditional interfaces using keyboard and mouse or trackball usually require a non-trivial amount of working space as well as a learning process. We present the design of EZ-Manipulator, a new 3D manipulation interface using smartphones that supports mobile, fast, and ambiguity-free interaction with 3D objects. Our system leverages the built-in multi-touch input and gyroscope sensor of smartphones to achieve 9 degrees-of-freedom axis-constrained manipulation and free-form rotation. Using EZ-Manipulator to manipulate objects in 3D is easy. The user merely has to perform intuitive singleor two-finger gestures and rotate the hand-held device to perform manipulations at fine-grained and coarse levels respectively.We further investigate the ambiguity in manipulation introduced by indirect manipulations using a multi-touch interface, and propose a dynamic virtual camera adjustment to effectively resolve the ambiguity. A preliminary study shows that our system has significant lower task completion time compared to conventional use of a keyboard–mouse interface, and provides a positive user experience to both novices and experts.

SynchroWatch is a one-handed interaction technique for smartwatches that uses rhythmic correlation between a user's thumb movement and on-screen blinking controls. Our technique uses magnetic sensing to track the synchronous extension and reposition of the thumb, augmented with a passive magnetic ring. The system measures the relative changes in the magnetic field induced by the required thumb movement and uses a time-shifted correlation approach with a reference waveform for detection of synchrony. We evaluated the technique during three distraction tasks with varying degrees of hand and finger movement: active walking, browsing on a computer, and relaxing while watching online videos. Our initial offline results suggest that intentional synchronous gestures can be distinguished from other movement. A second evaluation using a live implementation of the system running on a smartwatch suggests that this technique is viable for gestures used to respond to notifications or issue commands. Finally, we present three demonstration applications that highlight the technique running in real-time on the smartwatch.

Gesture-based systems allow users to interact with a virtual reality application in a natural way. Visual feedback for the gesture-based interaction technique has an impact on the performance and the hand instability making the manipulation of the object less precise. This paper investigated two new interaction techniques in a virtual environment. It describes the influence of natural and non-natural virtual feedback in the selection process using the GITDVR-G interaction technique, which consists of a grasping visual feedback. The GITDVR-G was evaluated in a virtual knee surgery training system. The results showed that it was effective in terms of the task completion time, and that the participants preferred the natural grasping visual feedback. Besides that, the precise manipulation in a newly-designed interaction technique (Precise GITDVR-G) was evaluated. The Precise GITDVR-G includes a normal manipulation mode and a precise manipulation mode that can be triggered by hand gestures. During the precise manipulation mode, an inset view will appear and move with the selected object to provide a better view to users, while the movements of the virtual hand are scaled down to improve the precision. Four different configurations of the precise manipulation technique were evaluated, and the results showed that the unimanual control method with an inset view performed better in terms of the task performance time and the subjective feedback. The finding suggested that the realistic virtual grasping visual feedback can be applied in a virtual hand interaction technique, and that the inset view feature is helpful in the precise manipulation.

In this work, we show how our open source accessibility software, the FaceSwitch, can help motor-impaired subjects to efficiently interact with a computer hands-free. The FaceSwitch enhances gaze interaction with video-based face gestures interaction. The emerging multimodal system allows for interaction with a user interface by means of gaze pointing for target selection and facial gestures for target-specific action commands. The FaceSwitch maps facial gestures to specific mouse or keyboard events such as: left mouse click, right mouse click, or page scroll down. Hence, facial gestures serve the purpose of mechanical switches. With this multimodal interaction paradigm, the user gazes at the object in the user interface with which it wants to interact and then triggers a target-specific action by performing a face gesture. Through a rigorous user study, we have obtained quantitative evidence that suggests our proposed interaction paradigm improves the performance of traditional accessibility options, such as gaze-only interaction or gaze with a single mechanical switch interaction while coming close in terms of speed and accuracy with traditional mouse-based interaction. We make the FaceSwitch software freely available to the community so the output of our research can help the target audience.

Interacting in 3D has considerable and growing importance today in several areas and most computer systems are equipped with both a mouse or touchscreen and one or more cameras that can be readily used for gesture-based interaction. Literature comparing mouse and free-hand gesture interaction, however, is still somewhat sparse in regards to user satisfaction, learnability and memorability, which can be particularly important attributes for applications in education or entertainment. In this paper, we compare the use of mouse-based interaction with free-hand bimanual gestures to translate, scale and rotate a virtual 3D object. We isolate object manipulation from its selection and focus particularly on those three attributes to test whether the related variables show significant differences when comparing both modalities. To this end, we integrate a gesture recognition system, Microsoft’s Kinect and an educational virtual atlas of anatomy, then design and perform an experiment with 19 volunteers, combining data from self-reported metrics and event logs. Our results show that, in this context, the difference is indeed significant and favors the use of gestures for most experimental tasks and evaluated attributes.

Audio-tactile charts have the potential to improve data analysis with tactile charts for blind people. Enhancing tactile charts with audio feedback can replace Braille labels and provide more structured information than pure tactile graphics. Current approaches lack especially in support of gestural interaction to develop useful interaction concepts for audio-tactile charts. Many approaches make use of non-standard hardware or are less mobile. That is why we investigated digital pens and their capability to increase data analysis with tactile charts. We compared two digital pens, in particular, the TipToi\(^{\textregistered }\) pen and the Neo SmartPen M1. First, we evaluated the implementation and feasibility of five basic gestures. While the TipToi\(^{\textregistered }\) is not suitable to support rich touch gestures, the Neo SmartPen showed in a pilot study good support of single-tap, double-tap as well as hold and line gestures. On that basis, we implemented the first prototype to demonstrate the potential of digital pens to support data analysis tasks with audio-tactile scatterplots. Afterwards, we evaluated the prototype in a pilot study with one participant. The study shows high indications for the usefulness of the presented system. The usage of the digital pen can improve the readability of a tactile chart. Our system provides audio-feedback for given tactile scatterplots in an accessible and automatic way. As a result, blind users were able to produce and use audio-tactile charts on their own by using an Android application and the Neo SmartPen.

National audience; Expert interaction techniques such as gestures or hotkeys are more efficient than traditional WIMP techniques because it is often faster to recall a command than to navigate to it. However, many users seem to be reluctant to switch to expert interaction. We hypothesize the cause might be the aversion of making errors. To test this, we designed two intermediate modes for the FastTap interaction technique, allowing quick confirmation of what the user has retrieved from memory, and quick adjustment if she has made an error. We investigated the impact of these modes and of various error costs in a controlled study (N=36). We found that participants adopted the intermediate modes, that these modes reduced error rate when error cost was high, and that they did not substantially change selection times. However, while it validates the design of our intermediate modes, we found no evidence of greater switch to memory-based interaction, suggesting that reducing the error rate is not sufficient to promote expert use of techniques.; Les techniques d'interaction expertes comme les vocabulaires gestuels ou les raccourcis clavier sont plus efficaces que les techniques WIMP traditionnelles. Il est en effet plus rapide de se rappeler une commande plutôt que de la retrouver dans des menus. Cependant, la plupart des utilisateurs semblent réticents à passer aux interactions qui se basent sur leur mémoire. Nous pensons que la cause pourrait être due à leur aversion à faire des erreurs. Pour tester cette hypothèse, nous avons conçu deux modes intermédiaires pour la technique d'interaction FastTap, qui permet de rapidement confirmer ce que l'utilisateur s'est rappelé de mémoire, et d'ajuster si une erreur a été faite. Nous avons étudié l'impact de ces deux modes intermédiaires et de différents coûts d'erreur dans une étude contrôlée (N=36). Nous avons trouvé que les participants ont adopté les modes intermédiaires, que ces modes réduisaient le taux d'erreur quand le coût de l'erreur était important, et qu'ils n'ont pas affecté de manière significative les temps de sélection. Cependant, bien que les résultats valident la conception de nos modes intermédiaires, nous n'avons pas trouvé de preuve sur un plus grand passage aux interactions qui se basent sur la mémoire. Cela suggère que réduire le taux d'erreur n'est pas suffisant pour promouvoir l'utilisation experte des techniques.

National audience; We present SmartCom, a combination of a smartphone and the traditional physical keyboard to enhance command selection by 1) eliminating occlusion, 2) reducing command discoverability time and 3) reducing homing cost. We performed a reachability study to evaluate the feasibility of gestures combining key press and phone touch. The results are promising and none of our participants found SmartCom usage difficult. In the future, we plan to design and evaluate novel interaction techniques for SmartCom.; Nous présentons SmartCom, une combinaison d'un smartphone et du clavier physique traditionnel pour améliorer la sélection des commandes en 1) éliminant l'occlusion, 2) réduisant le temps de découverte des commandes et 3) réduisant le coût de l'homing. Nous avons réalisé une étude d'accessibilité pour évaluer la faisabilité de gestes combinant l'appui sur les touches du clavier et le toucher sur le smartphone. Les résultats sont prometteurs et aucun de nos participants n'a trouvé l'utilisation de SmartCom difficile. Prochainement, nous prévoyons de concevoir et évaluer de nouvelles techniques d'interaction pour SmartCom.

Unlike retail stores, in which the user is forced to be physically present and active during restricted opening hours, online shops may be more convenient, functional and efficient. However, traditional online shops often have a narrow bandwidth for product visualizations and interactive techniques and lack a compelling shopping context. In this paper, we report a study on eliciting user-defined gestures for shopping tasks in an immersive VR (virtual reality) environment. We made a methodological contribution by providing a varied practice for producing more usable freehand gestures than traditional elicitation studies. Using our method, we developed a gesture taxonomy and generated a user-defined gesture set. To validate the usability of the derived gesture set, we conducted a comparative study and answered questions related to the performance, error count, user preference and effort required from end-users to use freehand gestures compared with traditional immersive VR interaction techniques, such as the virtual handle controller and ray-casting techniques. Experimental results show that the freehand-gesture-based interaction technique was rated to be the best in terms of task load, user experience, and presence without the loss of performance (i.e., speed and error count). Based on our findings, we also developed several design guidelines for gestural interaction.

Application that adopts collaborative system allows multiple users to interact with other users in the same virtual space either in Virtual Reality (VR) or Augmented Reality (AR). This paper aims to integrate the VR and AR space in a Collaborative User Interface that enables the user to cooperate with other users in a different type of interfaces in a single shared space manner. The gesture interaction technique is proposed as the interaction tool in both of the virtual spaces as it can provide a more natural gesture interaction when interacting with the virtual object. The integration of VR and AR space provide a cross-discipline shared data interchange through the network protocol of client-server architecture.

Mixed Reality (MR) is the next evolution of humans interacting with computer as MR can combine the physical environment and digital environment and making them coexist with each other [1]. Interaction is still a valid research area in MR, and this paper focuses on interaction rather than other research areas such as tracking, calibration, and display [2] because the current interaction technique still not intuitive enough to let the user interact with the computer. This paper explores the user interaction using gesture and speech interaction for 3D object manipulation in mixed reality environment. The paper explains the design stage that involves interaction using gesture and speech inputs to enhance user experience in MR workspace. After acquiring gesture input and speech commands, MR prototype is proposed to integrate the interaction technique using gesture and speech. The paper concludes with results and discussion.

KeySlide is a novel interaction technique to position the text cursor in text editors by refashioning standard physical keyboard used in desktop and laptop devices. The technique enables users to position the cursor at the desired location on text documents using natural and intuitive finger-sliding gestures over the keys of the keyboard. The natural mapping of the gesture to the action eliminates the learnability of the action, and minimises the disruption caused due to switching between the keyboard and the mouse. The design, implementation and evaluation of KeySlide is presented. While the theoretical simulation study found that KeySlide is 52% faster than standard keyboard shortcuts for cursor positioning, a comparison user study consisting of 18 participants, found that text editing performance using KeySlide is 19% slower than using keyboard shortcuts. The discrepancy is discussed.

We propose a one-handed interaction technique using cursor based on touch pressure to enable users to perform various single-touch gestures such as a tap, swipe, drag, and double-tap on unreachable targets. In the proposed technique, cursor mode is started by swiping from the bezel. Touch-down and touch-up events occur at the cursor position when users increase and decrease touch pressure, respectively. Since touch-down and touch-up event triggers are different but easily performed by just adjusting the touch pressure of the thumb from low to high or vice versa, the user can perform single-touch gestures at the cursor position with the thumb. To investigate the performance of the proposed technique, we conducted a pilot study; the results showed that the proposed technique is promising for one-handed interaction technique.

International audience; Relative pointing through using tactile mobile device (such as tablets of phones) on a large display is a viable interaction technique (that we call Pad in this paper) which permits accurate pointing. However, limited device size has consequences on interaction. Such systems are known to often require clutching, which degrades performances. We present E-Pad, an indirect relative pointing interaction technique which takes benefit of the mobile tactile surface combined with its surrounding space. A user can perform continuous relative pointing starting on the pad then continuing in the free space around the pad, within arm's reach. As a first step toward E-Pad, we first introduce extended continuous relative pointing gestures and conduct a preliminary study to determine how people move their hand around the mobile device. We then conduct an experiment that compares the performance of E-Pad and Pad. Our findings indicate that E-Pad is faster than Pad and decreases the number of clutches without compromising accuracy. Our findings also suggest an overwhelming preference for E-Pad.

We present a gesture-based user interface for smart TVs that employs deictic gestures to control the content displayed on the TV screen. Our interface implements an instance of the “Smart-Pockets” interaction technique, where links to digital content, in our case to users’ preferred television channels and shows, are stored inside users’ pockets and readily accessible with a mere pointing of the hand to those pockets. Pointing gestures to the pockets and towards the TV screen are detected using the Inertial Measurement Unit embedded in Myo, a smart armband. We discuss the ways in which our prototype opens new opportunities for hybrid, gesture- and pointing-based interactions for smart TVs as well as opportunities for designing interactions that take place at the periphery of user attention.

Three Dimensional (3D) interaction is the plausible human interaction inside a Virtual Environment (VE). The rise of the Virtual Reality (VR) applications in various domains demands for a feasible 3D interface. Ensuring immersivity in a virtual space, this paper presents an interaction technique where manipulation is performed by the perceptive gestures of the two dominant fingers; thumb and index. The two fingertip-thimbles made of paper are used to trace states and positions of the fingers by an ordinary camera. Based on the positions of the fingers, the basic interaction tasks; selection, scaling, rotation, translation and navigation are performed by intuitive gestures of the fingers. Without keeping a gestural database, the features-free detection of the fingers guarantees speedier interactions. Moreover, the system is user-independent and depends neither on the size nor on the color of the users’ hand. With a case-study project; Interactions by the Gestures of Fingers (IGF) the technique is implemented for evaluation. The IGF application traces gestures of the fingers using the libraries of OpenCV at the back-end. At the front-end, the objects of the VE are rendered accordingly using the Open Graphics Library; OpenGL. The system is assessed in a moderate lighting condition by a group of 15 users. Furthermore, usability of the technique is investigated in games. Outcomes of the evaluations revealed that the approach is suitable for VR applications both in terms of cost and accuracy.

Interacting with a smartphone using touch input and speech output is challenging for visually impaired people in mobile and public scenarios, where only one hand may be available for input (e.g., while holding a cane) and using the loudspeaker for speech output is constrained by environmental noise, privacy, and social concerns. To address these issues, we propose EarTouch, a one-handed interaction technique that allows the users to interact with a smartphone using the ear to perform gestures on the touchscreen. Users hold the phone to their ears and listen to speech output from the ear speaker privately. We report how the technique was designed, implemented, and evaluated through a series of studies. Results show that EarTouch is easy, efficient, fun and socially acceptable to use.

This paper presents a novel interaction technique to combine multiple inputs and output modalities with a 3D printed open disc which is tracked and augmented with a virtual information. Props have been used in virtual and augmented environments to provide tangible associations to virtual elements. When used in a larger immersive environment or where a user is walking around, users may not have the capability to set these objects/props down or are limited to the number of hands, so users are reduced to the capability of using only two props or controllers. In our technique, users can instead use their body to store and organize the virtual objects associated with tangible props by physically hanging them on or attaching them to the body. This type of interaction is well-suited for both immersive visualizations and immersive virtual environments. In this paper, we present the results of an experimental evaluation of our Body Prop Interaction technique for each type of immersive environment and corresponding task type, demonstrating more effective usage in each scenario than a joystick or gesture-based interaction.

This paper discusses our experiences, lessons learned, and future research directions in designing and running a field study of a motion-based gaming system for visitors of a local childrens museum. The system, named Virtual STEM Buddies, uses a large-screen kiosk to present minigames with interactive 3D content, such that the level of performance exhibited by participants indicates a level of understanding about the STEM concepts. The evolution of the system is presented, alongside evidence of improved usability and engagement throughout several prototyping iterations where the system has been used by thousands of visitors. We also describe a recently integrated mid-air free-hand interaction technique that facilitates selection and manipulation while staying accessible and intuitive to child visitors. Ultimately, we aim to learn how to best enable longitudinal interactions with the system that integrate virtual learning with the physically-rich museum learning environment.

Two-finger operations, such as rotation operations and zoom operations, are now standard touchscreen operations. However, these two-finger operations may pose an ergonomic problem when they are performed with one hand. FingerSkate is an interaction technique to mitigate the ergonomic problem of two-finger operations adopting the concept of gesture relaxation. A preliminary study showed that the FingerSkate technique was accepted well by the participants. In particular, a micro-analysis of finger movements during the study showed that FingerSkate was utilized well for large rotations. In a subsequent user study, the effects of two FingerSkate design options, Concurrency and Pivot, were examined. The Concurrency option had a statistically significant effect on performance and task workload for a docking task. The Pivot option had a statistically significant effect on task workload for a puzzle task. More participants preferred the sequential Concurrency option, and most participants preferred the ce...

The use of a physical controller like mouse, keyboard for human computer interaction hinders natural interface as there is a strong barrier between the user and computer. In this paper, we have designed a robust marker- less hand gesture recognition system which can efficiently track both static and dynamic hand gestures. Our system translates the detected gesture into actions such as opening websites and launching applications like VLC Player and PowerPoint. The dynamic gesture is used to shuffle through the slides in presentation. Our results show that an intuitive HCI can be achieved with minimum hardware requirements.

This article aims to assess the effect of embodied interaction on attention during the process of solving spatio-visual navigation problems. It presents a method that links operator's physical interaction, feedback, and attention. Attention is inferred through networks called Bayesian Attentional Networks (BANs). BANs are structures that describe cause-effect relationship between attention and physical action. Then, a utility function is used to determine the best combination of interaction modalities and feedback. Experiments involving five physical interaction modalities (vision-based gesture interaction, glove-based gesture interaction, speech, feet, and body stance) and two feedback modalities (visual and sound) are described. The main findings are: (i) physical expressions have an effect in the quality of the solutions to spatial navigation problems; (ii) the combination of feet gestures with visual feedback provides the best task performance.

This paper proposes a novel human-computer interaction system exploiting gesture recognition. It is based on the combined usage of an head-mounted display and a multi-modal sensor setup including also a depth camera. The depth information is used both to seamlessly include augmented reality elements into the real world and as input for a novel gesture-based interface. Reliable gesture recognition is obtained through a real-time algorithm exploiting novel feature descriptors arranged in a multi-dimensional structure fed to an SVM classifier. The system has been tested with various augmented reality applications including an innovative human-computer interaction scheme where virtual windows can be arranged into the real world observed by the user.

The virtual Experimental Advanced Superconducting Tokamak device (VEAST) is a very complicated 3D model, to interact with which, the traditional interaction devices are limited and inefficient. However, with the development of human-computer interaction (HCI), the hand gesture interaction has become a much popular choice in recent years. In this paper, we propose a real-time vision-based hand gesture interaction system for VEAST. By using one web camera, we can use our bare hand to interact with VEAST at a certain distance, which proves to be more efficient and direct than mouse. The system is composed of four modules: initialization, hand gesture recognition, interaction control and system settings. The hand gesture recognition method is based on codebook (CB) background modeling and open finger counting. Firstly, we build a background model with CB algorithm. Then, we segment the hand region by detecting skin color regions with “elliptical boundary model” in CbCr flat of YCbCr color space. Open finger which is used as a key feature of gesture can be tracked by an improved curvature-based method. Based on the method, we define nine gestures for interaction control of VEAST. Finally, we design a test to demonstrate effectiveness of our system.

We propose a new method to recognize gestures from sound waves. The main contribution of this paper is to recognize gestures based on the analysis of short-time Fourier transforms (STFT) using the Doppler effect to sense gestures. To do this, we generate an inaudible tone, which gets frequency-shifted when it reflects off moving objects like the hand. We measure this shift with the microphone to infer various gestures. Experimenting method and evaluating results by using the hand gestures of many different people to browse applications such as website, document and images in the browser on the computers in the classroom and library environment for accurate results. In addition, we describe the phenomena and recognition algorithm, demonstrate a variety of gestures, and present an informal evaluation on the robustness of this approach across Laptop device and people.

We describe the iterative design and evaluation of a geometric interaction technique for bare-hand mid-air virtual pottery. We model the shaping of a pot as a gradual and progressive convergence of the pot-profile to the shape of the user's hand represented as a point-cloud (PCL). Our pottery-inspired application served as a platform for systematically revealing how users use their hands to express the intent of deformation during a pot shaping process. Our approach involved three stages: (a) clutching by proximal-attraction, (b) shaping by proximal-attraction, and (c) shaping by grasp+motion. The design and implementation of each stage was informed by user evaluations of the previous stage. Our work evidently demonstrates that it is possible to enable users to express their intent for shape deformation without the need for a fixed set of gestures for clutching and deforming a shape. We found that the expressive capability of hand articulation can be effectively harnessed for controllable shaping by organizing the deformation process in broad classes of intended operations such as pulling, pushing, and fairing. After minimal practice with the pottery application, users could figure out their own strategy for reaching, grasping, and deforming the pot. Users particularly enjoyed using day-to-day physical objects as tools for shaping pots. Graphical abstractGrasp detection from Kernel density.Display Omitted HighlightsWe present iterative algorithm development for mid-air virtual pottery design.We extract grasp and motion using kernel-density of the point-cloud of hands.We use hand grasp and motion for classifying intent towards pot's deformation.Users can shape pots with hands as well as physical objects as tools.Evaluation shows increase in controllability in intent expression in pottery.

The emergence in computing and the latest hardware technologies realized the use of natural interaction with computers. Gesture-based interaction is one of the prominent fields of natural interactions. The recognition and application of hand gestures in virtual environments (VEs) need extensive calculations due to the complexities involved, which directly affect the performance and realism of interaction. In this paper, we propose a new interaction technique that uses single fingertip-based gestures for interaction with VEs. The objective of the study is to minimize the computational cost, increase performance, and improve usability. The interaction involves navigation, selection, translation, and release of objects. For this purpose, we propose a low-cost camera-based system that uses a colored fingertip for the fastest and accurate recognition of gestures. We also implemented the proposed interaction technique using the Leap Motion controller. We present a comparative analysis of the proposed system with the Leap Motion controller for gesture recognition and operation. A VE was developed for experimental purposes. Moreover, we conducted a comprehensive analysis of two different recognition setups including video camera and the Leap Motion sensor. The key parameters for analysis were task accuracy, interaction volume, update rate, and spatial distortion of accuracy. We used the Standard Usability Scale (SUS) for system usability analysis. The experiments revealed that camera implementation was found with good performance, less spatial distortion of accuracy, and large interaction volume as compared to the Leap Motion sensor. We also found the proposed interaction technique highly usable in terms of user satisfaction, user-friendliness, learning, and consistency.

Augmented Reality (AR) is the technology that augments the real world by using virtual information of 3D objects overlaid on a view of the real environment. This will create an immersive and intuitive experience. As handheld devices are widely used and now has climb with increasing demand of specifications, its interaction based on touch is a natural and appealing input style for the application of AR. Furthermore, most heuristic studies of interaction in AR usually focus on interactions with AR Target close to the user, generally within arm’s reach. As the user’s move farther away, the effectiveness and usability of the interaction modalities may be different. This study explores handheld AR interaction using real hand gesture at a distance in a room-scale setup. Our aim is to investigate the effectiveness of performing selection while being far away from the 3D object and performing selection task towards object that is being occluded. As object manipulation is one of the key features to explore for mobile handheld AR, we hope our study can give some contribution towards its practical use, especially for real-world assembly structure. Hence, our paper proposes finger ray interaction technique for real hand in handheld AR interface to work for selecting objects at distances ranging from 3 feet to 8 feet and when the object being occluded ranging from 20% to 80% occluded.