Search

Human-mobile interaction is aimed at facilitating interaction with the smartphone devices. The conventional way to interact with mobile devices is through manual input where most of the applications are made assuming that the end user has full control over their hand movements. However, this assumption excludes people who are unable to use their hands or have suffered limb damage. In this paper, we proposed a nose and teeth based interaction system, which allows the users to control their mobile devices completely hands free. The proposed system uses the front facing camera of the smartphone to track the position of the nose for cursor control on the smartphone screen. The system detects teeth for performing the touch screen events such as tap, scroll, long press, and drag. Viola-Jones algorithm is used to detect the face and teeth based on the Haar features. After detecting the face, the nose position is calculated and tracked continuously using Lucas Kanade’s method for optical flow estimation. All the touch screen events have been implemented in the system so that the user can execute all the operations of the smartphone. To evaluate the performance and the effect of (smartphone) device type on the execution time, the proposed system was installed in 3 smartphone devices and 7 trials for each device were performed by 3 different able-bodied elderly persons. The result shows a significant success rate for the detection of nose and teeth, and for the execution of the operations. The execution time of each operation slightly varies by 0.72s on average because of the configuration of the smartphones.

We present ScraTouch, an interaction technique using fingernails, as a new input modality by leveraging capacitive touch sensing. Differentiating between fingertip and fingernail touches requires only tens of milliseconds worth of shunt current data from unmodified capacitive touch surfaces, thus requires no hardware modification. ScraTouch is simple but practical technique for command invocation and mode switching. An evaluation using a point-and-select task on a touchpad showed that although the switching between the finger and nail in ScraTouch required a little more time compared with the baseline (finger touching without mode switching), in overall the operations, ScraTouch was just as fast as the baseline, and on average, 29 % faster than a long press with 500-ms threshold. We also confirmed that setting a simple threshold on the measured shunt current for recognition works robustly across users (97 % accuracy).

Computer-assisted orthopedic surgery (CAOS) and augmented reality provides guidance for surgical environments in minimally invasive procedures. However, the surgeon’s interaction with those systems has limitations, because the surgical environment must be kept sterile. Two touch-less interaction techniques were evaluated within this research work, a vision-based technique used by META® glasses and electromyography-based used by MYO® arm-band. The research aim was to establish the most appropriate one touch-less interaction technique to manipulate a CAOS system. Usability was evaluated through an experiment with 4 orthopedic surgeons and 47 undergraduate students, in a simulated setting. The results suggest that both techniques can be relevant and useful in a surgical environment. However, the MYO was more efficient and possibly more effective to perform the manipulation tasks than META. In addition, the MYO arm-band was perceived as further satisfactory compared to META glasses and shown an improved overall behaviour in the CAOS system manipulations tasks.

Head-operated computer accessibility tools (CATs) are useful solutions for the ones with complete head control; but when it comes to people with only reduced head control, computer access becomes a very challenging task since the users depend on a single head-gesture like a head nod or a head tilt to interact with a computer. It is obvious that any new interaction technique based on a single head-gesture will play an important role to develop better CATs to enhance the users’ self-sufficiency and the quality of life. Therefore, we proposed two novel interaction techniques namely HeadCam and HeadGyro within this study. In a nutshell, both interaction techniques are based on our software switch approach and can serve like traditional switches by recognizing head movements via a standard camera or a gyroscope sensor of a smartphone to translate them into virtual switch presses. A usability study with 36 participants (18 motor-impaired, 18 able-bodied) was also conducted to collect both objective and subjective evaluation data in this study. While HeadGyro software switch exhibited slightly higher performance than HeadCam for each objective evaluation metrics, HeadCam was rated better in subjective evaluation. All participants agreed that the proposed interaction techniques are promising solutions for computer access task.

International audience; Rotating 3D objects on desktop computers with a mouse or a trackpad is a notoriously difficult task, especially for novice users. Techniques relying on a “virtual trackball” have been proposed in the literature and these continue to be used in most 3D software. While several studies were conducted to compare the performance of these techniques, none was focused on the intrinsic parameter of the radius of the virtual control sphere of the trackball. We present the results of a controlled study to investigate the influence of the radius of the control sphere on the performance and behavior of users in a 3D docking task. Surprisingly, the results do not suggest a significant effect of the size of the virtual control sphere on user performance. However, an analysis of user behavior suggests that it influences the user’s strategy for how they interact with the virtual trackball.; La rotation d’objets 3D sur ordinateurs de bureau avec une souris ou un pavé tactile est une tâche connue pour sa difficulté, en particulier pour les utilisateurs débutants. Différentes techniques de sphères virtuelles ont été proposées dans la littérature pour faciliter cette tâche et ces techniques continuent d’être utilisées dans la plupart des logiciels 3D. Différentes études ont été réalisées dans la littérature afin de comparer la performance de ces techniques. Cependant aucune ne s’est intéressée à un paramètre intrinsèque de ces techniques qui est le rayon de la sphère virtuelle de contrôle. Nous présentons dans cet article les résultats d’une étude contrôlée dans le but d’étudier l’influence du rayon de la sphère virtuelle sur les performances et le comportement des utilisateurs dans une tâche d’ancrage 3D. Les résultats de cette étude n’ont pas montré d’effet significatif de la taille de sphère virtuelle de contrôle sur les performances des utilisateurs. Cependant une analyse du comportement des utilisateurs montre que la taille de la sphère de contrôle peut influencer la stratégie des participants.

International audience; We advocate for the usage of hotkeys on touch-based devices by capitalising on soft keyboards through four studies. First, we evaluated visual designs and recommended icons with command names for novices while letters with command names for experts. Second, we investigated the discoverability by asking crowdworkers to use our prototype, with some tasks only doable upon successfully discovering the technique. Discovery rates were high regardless of conditions that vary the familiarity and saliency of modifier keys. However, familiarity with desktop hotkeys boosted discoverability. Our third study focused on how prior knowledge of hotkeys could be leveraged and resulted in a 5% selection time improvement and identified the role of spatial memory in retention. Finally, we compared our soft keyboard layout with a grid layout similar to FastTap. The latter offered a 12-16% gain on selection speed, but at a high cost in terms of screen estate and low spatial stability.

Virtual reality technologies are evolving fast with the development of headsets that have increased display resolutions and motion tracking capabilities. These headsets are associated with controllers with many degrees of freedom that offer the possibility to improve and develop new interaction techniques. Indeed, interaction in these immersive environments is often more laborious than in reality. In this thesis, we focused on the improvement of the selection task and the development of techniques dedicated to the control of facial expressions.For the 3D object selection task, the virtual hand and Raycasting are the most frequently used techniques. Although Raycasting allows selecting objects out of reach, the selection is more difficult when the objects are far away, small or partially occluded. We have highlighted how to reduce the selection errors of Raycasting by using an adapted filtering.We also developed RayCursor, an improvement of Raycasting that relies on adding a cursor that the user can move along the ray. This cursor allows Raycasting to be combined with proximity-based pointing facilitation techniques. We studied a set of visual feedbacks adapted to the visualization and selection of nearby targets. A series of controlled experiments highlighted the benefits of RayCursor over a set of reference techniques from the literature.For the control of facial expressions, we have proposed a set of non-isomorphic techniques to overcome the limitations of the isomorphic techniques proposed in the literature, which rely on the detection or estimation of the users' real expressions to transpose them to their avatar. It is difficult to control precisely expressions and it is impossible for users to control an expression different from the expression on their face. Our techniques are based on the decomposition of facial expression control into subtasks that we formalize in a design space: facial expression selection, intensity and duration control. Controlled experiments allowed us to isolate the most relevant techniques to perform each of the subtasks, in order to design a technique that we call EmoRayE, that was validated in an ecological experiment.; Les technologies de réalité virtuelle connaissent un essor sans précédent par le développement de casques qui possèdent des résolutions d'affichage et des capacités de suivi du mouvement inédits. Ces casques sont associés à des contrôleurs dotés de nombreux degrés de liberté qui offrent la possibilité d'améliorer et de développer de nouvelles techniques d'interaction. L'interaction dans ces environnements immersifs reste en effet bien souvent plus laborieuse que dans la réalité. Nous nous sommes concentrés dans cette thèse sur l'amélioration de la tâche de sélection et le développement de techniques dédiées au contrôle d'expressions faciales.Pour la tâche de sélection d’objets 3D, la main virtuelle et le Raycasting sont les techniques les plus fréquemment utilisées. Bien que Raycasting permette de sélectionner des objets hors de portée, la sélection est d'autant plus difficile que les objets sont éloignés, petits ou partiellement occultés. Nous avons mis en évidence comment réduire les erreurs de sélection de Raycasting par l'utilisation d'un filtrage adapté. Nous avons également développé RayCursor, une amélioration de Raycasting qui repose sur l'ajout d'un curseur que l’utilisateur peut déplacer le long du rayon. Ce curseur permet de combiner Raycasting avec des techniques de facilitation du pointage par proximité. Nous avons étudié un ensemble de retours visuels adaptés à la visualisation et la sélection de cibles à proximité. Une série d'expériences contrôlées a permis de mettre en évidence les bénéfices de RayCursor par rapport à un ensemble de techniques de référence de la littérature.Pour le contrôle d'expressions faciales, nous avons proposé un ensemble de techniques non isomophiques pour dépasser les limitations des techniques isomorphiques proposées dans la littérature, qui reposent sur la détection ou l'estimation de l'expression réelle de l'utilisateur pour la transposer à son avatar. Le contrôle précis d'expressions est difficile et il est impossible à l'utilisateur de contrôler une expression autre que celle de son visage. Les techniques que nous proposons reposent sur une décomposition du contrôle d'expressions faciales en sous-tâches que nous formalisons dans un espace de conception : la sélection de l’expression faciale, le contrôle de son intensité et de sa durée. Des expériences contrôlées ont permis d'isoler les techniques les plus pertinentes pour réaliser chacune des sous-tâches, afin de concevoir une technique que nous appelons EmoRayE. Celle-ci a été validée dans une expérience écologique.

International audience; The control of an avatar’s facial expressions in virtual reality is mainly based on the automated recognition and transposition of the user’s facial expressions. These isomorphic techniques are limited to what users can convey with their own face and have recognition issues. To overcome these limitations, non-isomorphic techniques rely on interaction techniques using input devices to control the avatar’s facial expressions. Such techniques need to be designed to quickly and easily select and control an expression, and not disrupt a main task such as talking. We present the design of a set of new non- isomorphic interaction techniques for controlling an avatar facial expression in VR using a standard VR controller. These techniques have been evaluated through two controlled experiments to help designing an interaction technique combining the strengths of each approach. This technique was evaluated in a final ecological study showing it can be used in contexts such as social applications.

Future augmented reality (AR) glasses may provide pervasive and continuous access to everyday information. However, it remains unclear how to address the issue of virtual information overlaying and occluding real-world objects and information that are of interest to users. One approach is to keep virtual information sources inactive until they are explicitly requested, so that the real world remains visible. In this research, we explored the design of interaction techniques with which users can activate virtual information sources in AR. We studied this issue in the context of Glanceable AR, in which virtual information resides at the periphery of the user’s view. We proposed five techniques and evaluated them in both sitting and walking scenarios. Our results demonstrate the usability, user preference, and social acceptance of each technique, as well as design recommendations to achieve optimal performance. Our findings can inform the design of lightweight techniques to activate virtual information displays in future everyday AR interfaces.

As smartphone screens have grown in size, single-handed use has become more cumbersome. Interactive targets that are easily seen can be hard to reach, particularly notifications and upper menu bar items. Users must either adjust their grip to reach distant targets, or use their other hand. In this research, we show how gaze estimation using a phone’s user-facing camera can be paired with IMU-tracked motion gestures to enable a new, intuitive, and rapid interaction technique on handheld phones. We describe our proof-of-concept implementation and gesture set, built on state-of-the-art techniques and capable of self-contained execution on a smartphone. In our user study, we found a mean euclidean gaze error of 1.7 cm and a seven-class motion gesture classification accuracy of 97.3%.

In order to satisfy users’ information needs while incurring minimum interference to their ongoing activities, previous studies have proposed using Optical Head-mounted Displays (OHMDs) with different input techniques. However, it is unclear how these techniques compare against one another in terms of being comfortable and non-intrusive to a user’s everyday tasks. Through a wizard-of-oz study, we thus compared four subtle interaction techniques (feet, arms, thumb-index-fingers, and teeth) in three daily hands-busy tasks under different settings (giving a presentation–sitting, carrying bags–walking, and folding clothes–standing). We found that while each interaction technique has its niche, thumb-index-finger interaction has the best overall balance and is most preferred as a cross-scenario subtle interaction technique for smart glasses. We provide further evaluation of thumb-index-finger interaction with an in-the-wild study with 8 users. Our results contribute to an enhanced understanding of user preferences for subtle interaction techniques with smart glasses for everyday use.

In-situ self-reporting on smartwatches has been widely-used to collect data about participants’ experience. Yet it places a burden on participants by requiring immediate response in context. Such user burden can be studied through users’ performance and perceptions. This paper evaluates six interaction techniques to study user burden of in-situ self-reporting from both performance and perception aspects under three simulated scenarios (i.e., gaming, social chatting, and walking) through an in-lab experiment with twenty-four participants. Findings show that user performance was not always aligned with their perceived burden owing to factors such as users’ acceptable interaction technique features for self-reporting, the context of use, and users’ acquired skills over time. This study also sheds lights onto how user burden is experienced across different stages of a self-reporting microinteraction. Reflecting on findings, we discuss the implications for understanding and minimizing user burden related to in-situ self-reporting and as well as other types of microinteractions.

As the range of handheld, mobile and desktop devices expands and worldwide demand for collaborative application tools increases, there is a growing need for higher speed impromptu cross-device application sharing to keep up with workplace requirements for on-site or remote collaborations. To address this, we have developed CamCutter, a cross-device interaction technique enabling a user to quickly select and share an application running on another screen using the camera of a handheld device. This technique can accurately identify the targeted application on a display using our adapted computer vision algorithm, system architecture and software implementation, allowing impromptu real-time and synchronized application sharing between devices. For desktop and meeting room set-ups, we performed a technical evaluation, measuring accuracy and speed of migration. For a single-user reading task and a collaborative composition task, we carried out a user study comparing our technique with commercial screen sharing applications. The results of this study showed both higher performance and preference for our system. Finally, we discuss CamCutter’s limitations and present insights for future vision-based cross-device application sharing.

Gesture-based interfaces offer a suitable platform for interactions in Virtual Environments (VE). However, the difficulties involved in learning and making of distinct gestures affect the performance of an interactive system. By incorporating computer vision in Virtual Reality (VR), this paper presents an intuitive interaction technique where the states and positions of eyes are traced for interaction. With comparatively low cognitive load, the technique offers an easy to use interaction solution for VR applications. Unlike other gestural interfaces, interactions are performed in distinct phases where transition from one phase to another is enacted with simple blink of eyes. In an attained phase, interaction along an arbitrary axis is performed by a perceptive gesture of head; rolling, pitching or yawing. To follow the trajectory of eyes in real time, coordinates mapping is performed dynamically. The proposed technique is implemented in a case-study project; EBI (Eyes Blinking based Interaction). In the EBI project, real time detection and tracking of eyes are performed at the back-end. At the front-end, virtual scene is rendered accordingly by using the OpenGL library. To assess accuracy, usability and cognitive load of the proposed technique, the EBI project is evaluated 280 times in three different evaluation sessions. With an ordinary camera, an average accuracy of 81.4% is achieved. However, assessment made by using a high-quality camera revealed that accuracy of the system could be raised to a higher level. As a whole, findings of the evaluations support applicability of the technique in the emerging domains of VR.

Speech input, such as voice assistant and voice message, is an attractive interaction option for mobile users today. However, despite its popularity, there is a use limitation for smartphone speech input: users need to press a button or say a wake word to activate it before use, which is not very convenient. To address it, we match the motion that brings the phone to mouth with the user's intention to use voice input. In this paper, we present ProxiTalk, an interaction technique that allows users to enable smartphone speech input by simply moving it close to their mouths. We study how users use ProxiTalk and systematically investigate the recognition abilities of various data sources (e.g., using a front camera to detect facial features, using two microphones to estimate the distance between phone and mouth). Results show that it is feasible to utilize the smartphone's built-in sensors and instruments to detect ProxiTalk use and classify gestures. An evaluation study shows that users can quickly acquire ProxiTalk and are willing to use it. In conclusion, our work provides the empirical support that ProxiTalk is a practical and promising option to enable smartphone speech input, which coexists with current trigger mechanisms.

Brainwaves can be used as auxiliary inputs in the design of game mechanics for attention training games. Although attention training games are generally developed in the single-player mode, multi-player games can be more engaging and fun to play. This paper proposes a new interaction technique between players in multi-player car racing games, in which a novel competitive reinforcement approach is used to regulate difficulty parameters using brainwaves of players. The results of our experiments suggest that the competitive reinforcement is more effective to increase attention level as compared to positive and negative reinforcement conditions.

Biomolecules, such as RNA, DNA, proteins and polysaccharides, are at the heart of fundamental cellular processes. These molecules differ greatly with each other in terms of their structures and functions. However, in the midst of the diversity of biomolecules is the unifying feature that they interact with each other to execute a viable biological system. Interactions of biomolecules are critical for cells to survive and replicate, for food metabolism to produce energy, for antibiotics and vaccines to function, for spreading of diseases and for every other biological process. An improved understanding of these interactions is crucial for studying how cells and organs function, to appreciate how diseases are caused and how infections occur, with infinite implications in medicine and therapy. Many biochemical and biophysical techniques are currently being employed to study biomolecular interactions. Microscale thermophoresis (MST) is a relatively new biophysical technique that can provide powerful insight into the interactions of biomolecules and is quickly being adopted by an increasing number of researchers worldwide. This article provides a brief description of principles underpinning the MST process, in addition to benefits and limitations.

There is a significant change in technology, virtual reality (VR) has been used in this article as the simplest way of visual activity and interaction technique which provides visual nodal options supported by the automated lip-reading technology. With this advancement in the technology, the state of the human can be identified and captured through the lip movements. Deep Learning is used to analyze the real time thinking of the human visual choices. Using image processing technique, virtual reality technology is used to identify the driver's visual features and evaluate the critical time thinking. The ancient lip-reading recognition system, the need for responsive applications is difficult to fulfill. Deep Learning is now the emerging technique of artificial intelligence which acts as a normal human brain with the thinking capability. It consists of different layers which are used to evaluate the details like neurons in brain. In this article, the surface area of the lip is taken as the key element or the key feature of the lip movement. The horizontal distance and the vertical distance of the lips are used to calculate the surface area. This surface area is then used to estimate some parameter and store in the database. Based on the results, the accuracy is more than 85%. Therefore, our proposed system is more effective and more efficient than other traditional methods.

This paper proposes a novel text interface using deep learning in a mobile platform environment and presents the English language teaching applications created based on our interface. First, an interface for handwriting texts is designed with a simple structure based on a touch-based input method of mobile platform applications. This input method is easier and more convenient than the existing graphical user interface (GUI), in which menu items such as buttons are selected repeatedly or step by step. Next, an interaction that intuitively facilitates a behavior and decision making from the input text is proposed. We propose an interaction technique that recognizes a text handwritten on the text interface through the Extended Modified National Institute of Standards and Technology (EMNIST) dataset and a convolutional neural network (CNN) model and connects the text to a behavior. Finally, using the proposed interface, we create English language teaching applications that can effectively facilitate learning alphabet writing and words using handwriting. Then, the satisfaction regarding the interface during the educational process is analyzed and verified through a survey experiment with users.

Virtual Reality (VR) headsets can open opportunities for users to accomplish complex tasks on large virtual displays using compact and portable devices. However, interacting with such large virtual displays using existing interaction techniques might cause fatigue, especially for precise manipulation tasks, due to the lack of physical surfaces. To deal with this issue, we explored the design of VXSlate, an interaction technique that uses a large virtual display as an expansion of a tablet. We combined a user’s head movements as tracked by the VR headset, and touch interaction on the tablet. Using VXSlate, a user head movements positions a virtual representation of the tablet together with the user’s hand, on the large virtual display. This allows the user to perform fine-tuned multi-touch content manipulations. In a user study with seventeen participants, we investigated the effects of VXSlate on users in problem-solving tasks involving content manipulations at different levels of difficulty, such as translation, rotation, scaling, and sketching. As a baseline for comparison, off-the-shelf touch-controller interactions were used. Overall, VXSlate allowed participants to complete the task with completion times and accuracy that are comparable to touch-controller interactions. After an interval of use, VXSlate significantly reduced users’ time to perform scaling tasks in content manipulations, as well as reducing perceived effort. We reflected on the advantages and disadvantages of VXSlate in content manipulation on large virtual displays and explored further applications within the VXSlate design space.