Search

There is increased interest in using generative AI to create 3D spaces for Virtual Reality (VR) applications. However, today's models produce artificial environments, falling short of supporting collaborative tasks that benefit from incorporating the user's physical context. To generate environments that support VR telepresence, we introduce SpaceBlender, a novel pipeline that utilizes generative AI techniques to blend users' physical surroundings into unified virtual spaces. This pipeline transforms user-provided 2D images into context-rich 3D environments through an iterative process consisting of depth estimation, mesh alignment, and diffusion-based space completion guided by geometric priors and adaptive text prompts. In a preliminary within-subjects study, where 20 participants performed a collaborative VR affinity diagramming task in pairs, we compared SpaceBlender with a generic virtual environment and a state-of-the-art scene generation framework, evaluating its ability to create virtual spaces suitable for collaboration. Participants appreciated the enhanced familiarity and context provided by SpaceBlender but also noted complexities in the generative environments that could detract from task focus. Drawing on participant feedback, we propose directions for improving the pipeline and discuss the value and design of blended spaces for different scenarios.

This paper introduces holographic cross-device interaction, a new class of remote cross-device interactions between local physical devices and holographically rendered remote devices. Cross-device interactions have enabled a rich set of interactions with device ecologies. Most existing research focuses on co-located settings (meaning when users and devices are in the same physical space) to achieve these rich interactions and affordances. In contrast, holographic cross-device interaction allows remote interactions between devices at distant locations by providing a rich visual affordance through real-time holographic rendering of the device's motion, content, and interactions on mixed reality head-mounted displays. This maintains the advantages of having a physical device, such as precise input through touch and pen interaction. Through holographic rendering, not only can remote devices interact as if they are co-located, but they can also be virtually augmented to further enrich interactions, going beyond what is possible with existing cross-device systems. To demonstrate this concept, we developed HoloDevice, a prototype system for holographic cross-device interaction using the Microsoft Hololens 2 augmented reality headset. Our contribution is threefold. First, we introduce the concept of holographic cross-device interaction. Second, we present a design space containing three unique benefits, which include: (1) spatial visualization of interaction and motion, (2) rich visual affordances for intermediate transition, and (3) dynamic and fluid configuration. Last we discuss a set of implementation demonstrations and use-case scenarios that further explore the space.

Today's video-conferencing tools support a rich range of professional and social activities, but their generic meeting environments cannot be dynamically adapted to align with distributed collaborators' needs. To enable end-user customization, we developed BlendScape, a rendering and composition system for video-conferencing participants to tailor environments to their meeting context by leveraging AI image generation techniques. BlendScape supports flexible representations of task spaces by blending users' physical or digital backgrounds into unified environments and implements multimodal interaction techniques to steer the generation. Through an exploratory study with 15 end-users, we investigated whether and how they would find value in using generative AI to customize video-conferencing environments. Participants envisioned using a system like BlendScape to facilitate collaborative activities in the future, but required further controls to mitigate distracting or unrealistic visual elements. We implemented scenarios to demonstrate BlendScape's expressiveness for supporting environment design strategies from prior work and propose composition techniques to improve the quality of environments., Comment: ACM UIST 2024

Two-dimensional pin array tactile displays enable access to tactile graphics that are important for the education of students with visual impairments. Due to their prohibitive cost, limited access, and limited research within HCI, the rules to design graphical primitives on these low-resolution tactile displays are unclear. In this paper, eight tactile readers with visual impairments qualitatively evaluate the implementation of Pixel Art to create tactile graphical primitives on a pin array display. Every pin of the pin array is assumed to be a pixel on a pixel grid. Our findings suggest that Pixel Art tactile graphics on a pin array are clear and comprehensible to tactile readers, positively confirming its use to design basic tactile shapes and line segments. The guidelines provide a framework to create tactile media which implies that the guidelines can be used to downsize basic shapes for refreshable pin-array displays., Comment: 25 pages, 10 figures. To appear in DIS'23 Designing Interactive Systems Conference, July 10 to 14, 2023, Pittsburgh, PA, USA

This paper contributes to a taxonomy of augmented reality and robotics based on a survey of 460 research papers. Augmented and mixed reality (AR/MR) have emerged as a new way to enhance human-robot interaction (HRI) and robotic interfaces (e.g., actuated and shape-changing interfaces). Recently, an increasing number of studies in HCI, HRI, and robotics have demonstrated how AR enables better interactions between people and robots. However, often research remains focused on individual explorations and key design strategies, and research questions are rarely analyzed systematically. In this paper, we synthesize and categorize this research field in the following dimensions: 1) approaches to augmenting reality; 2) characteristics of robots; 3) purposes and benefits; 4) classification of presented information; 5) design components and strategies for visual augmentation; 6) interaction techniques and modalities; 7) application domains; and 8) evaluation strategies. We formulate key challenges and opportunities to guide and inform future research in AR and robotics., Comment: CHI 2022

Exploring and manipulating complex virtual objects is challenging due to limitations of conventional controllers and free-hand interaction techniques. We present the TanGi toolkit which enables novices to rapidly build physical proxy objects using Composable Shape Primitives. TanGi also provides Manipulators allowing users to build objects including movable parts, making them suitable for rich object exploration and manipulation in VR. With a set of different use cases and applications we show the capabilities of the TanGi toolkit, and evaluate its use. In a study with 16 participants, we demonstrate that novices can quickly build physical proxy objects using the Composable Shape Primitives, and explore how different levels of object embodiment affect virtual object exploration. In a second study with 12 participants we evaluate TanGi's Manipulators, and investigate the effectiveness of embodied interaction. Findings from this study show that TanGi's proxies outperform traditional controllers, and were generally favored by participants., Comment: 10 pages, 11 figures

Designing for situational awareness could lead to better solutions for disabled people, likewise, exploring the needs of disabled people could lead to innovations that can address situational impairments. This in turn can create non-stigmatising assistive technology for disabled people from which eventually everyone could benefit. In this paper, we investigate the potential for advanced haptics to compliment the graphical user interface of mobile devices, thereby enhancing user experiences of all people in some situations (e.g. sunlight interfering with interaction) and visually impaired people. We explore technical solutions to this problem space and demonstrate our justification for a focus on the creation of kinaesthetic force feedback. We propose initial design concepts and studies, with a view to co-create delightful and expressive haptic interactions with potential users motivated by scenarios of situational and permanent impairments., Comment: Presented at the CHI'19 Workshop: Addressing the Challenges of Situationally-Induced Impairments and Disabilities in Mobile Interaction, 2019 (arXiv:1904.05382)

We introduce Deep Thermal Imaging, a new approach for close-range automatic recognition of materials to enhance the understanding of people and ubiquitous technologies of their proximal environment. Our approach uses a low-cost mobile thermal camera integrated into a smartphone to capture thermal textures. A deep neural network classifies these textures into material types. This approach works effectively without the need for ambient light sources or direct contact with materials. Furthermore, the use of a deep learning network removes the need to handcraft the set of features for different materials. We evaluated the performance of the system by training it to recognise 32 material types in both indoor and outdoor environments. Our approach produced recognition accuracies above 98% in 14,860 images of 15 indoor materials and above 89% in 26,584 images of 17 outdoor materials. We conclude by discussing its potentials for real-time use in HCI applications and future directions., Comment: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems

We present RealPen, an augmented stylus for capacitive tablet screens that recreates the physical sensation of writing on paper with a pencil, ball-point pen or marker pen. The aim is to create a more engaging experience when writing on touch surfaces, such as screens of tablet computers. This is achieved by re-generating the friction-induced oscillation and sound of a real writing tool in contact with paper. To generate realistic tactile feedback, our algorithm analyses the frequency spectrum of the friction oscillation generated when writing with traditional tools, extracts principal frequencies, and uses the actuator's frequency response profile for an adjustment weighting function. We enhance the realism by providing the sound feedback aligned with the writing pressure and speed. Furthermore, we investigated the effects of superposition and fluctuation of several frequencies on human tactile perception, evaluated the performance of RealPen, and characterized users' perception and preference of each feedback type., Comment: Proceedings of the 29th Annual Symposium on User Interface Software and Technology (UIST '16)

The ability of sensing breathing is becoming an increasingly important function for technology that aims at supporting both psychological and physical wellbeing. We demonstrate ThermSense, a new breathing sensing platform based on smartphone technology and low-cost thermal camera, which allows a user to measure his/her breathing pattern in a contact-free manner. With the designed key functions of Thermal Voxel Integration-based breathing estimation and respiration variability spectrogram (RVS, bi-dimensional representation of breathing dynamics), the developed platform provides scalability and flexibility for gathering respiratory physiological measurements ubiquitously. The functionality could be used for a variety of applications from stress monitoring to respiration training., Comment: The Seventh AAAC International Conference on Affective Computing and Intelligent Interaction Workshops and Demos

The ability to monitor respiratory rate is extremely important for medical treatment, healthcare and fitness sectors. In many situations, mobile methods, which allow users to undertake every day activities, are required. However, current monitoring systems can be obtrusive, requiring users to wear respiration belts or nasal probes. Recent advances in thermographic systems have shrunk their size, weight and cost, to the point where it is possible to create smart-phone based respiration rate monitoring devices that are not affected by lighting conditions. However, mobile thermal imaging is challenged in scenes with high thermal dynamic ranges. This challenge is further amplified by general problems such as motion artifacts and low spatial resolution, leading to unreliable breathing signals. In this paper, we propose a novel and robust approach for respiration tracking which compensates for the negative effects of variations in the ambient temperature and motion artifacts and can accurately extract breathing rates in highly dynamic thermal scenes. It has three main contributions. The first is a novel Optimal Quantization technique which adaptively constructs a color mapping of absolute temperature to improve segmentation, classification and tracking. The second is the Thermal Gradient Flow method that computes thermal gradient magnitude maps to enhance accuracy of the nostril region tracking. Finally, we introduce the Thermal Voxel method to increase the reliability of the captured respiration signals compared to the traditional averaging method. We demonstrate the extreme robustness of our system to track the nostril-region and measure the respiratory rate in high dynamic range scenes., Comment: Vol. 8, No. 10, 1 Oct 2017, Biomedical Optics Express 4480 - Full abstract can be found in this journal article (due to limited word counts of 'arXiv abstract')