Search

Augmented Reality (AR) displays show virtual information spatially in the physical world. For example, in the surgical domain, AR displays can overlay medical imaging on top of the patient to help surgeons distinguish healthy tissue from cancerous tissue. This affordance of adding visual information near the area of action has led to a gamut of AR applications with one caveat: their design assumes a flawless alignment between the virtual information and its physical context. If this alignment fails, the system might mislead users by presenting the virtual information out of context.Fortunately, technical advancements in tracking, calibration, and optics have helped improve and create a more seamless augmented reality experience; however, even in these improved systems, alignment failure will lead to user-task performance errors. In my dissertation, I argue that AR interface design needs to account for such misalignments and limitations of AR technology. First, I investigated the case of a situated checklist for cardiopulmonary resuscitation (HoloCPR), which helped deliver accurate guidance even in the presence of tracking errors. Then, I investigated a telementoring system for trauma procedures (ARTEMIS) which helped remote surgeons communicate with novices despite a constant registration error. Finally, I showed that slight optical deviation can drastically impact overall user-task performance and cognitive workload, even in an idealized environment with perfect tracking and calibration.I sum the findings of this final piece, along with the systems designed and developed in this dissertation, to point in the direction of a new type of AR interface building block: contextual aids and how they may have significant implications for AR interface design for surgical procedures, paving the way for the development and integration of more advanced AR systems in surgical practice.

With the increasing adoption of Augmented Reality/Virtual Reality (AR/VR - XR) systems, security, privacy, and quality concerns attract attention from both academia and industry.We identify four key problems in XR applications and make contributions respectively.For XR Quality of experience: AR applications suffer from spatial inconsistency where virtual objects "drift or "jump" in space. We develop an automated method to evaluate the spatial inconsistency of AR applications. Web-based XR suffers from long latency in downloading virtual content to be displayed on a web browser. We develop a technique to divide up and prioritize virtual objects to be downloaded in a manor to minimize the time to first correct image. For XR Security: Head mounted displays track the user's head pose at all times in order to render content correctly. We show that these head poses can be used to help infer user hand-typed words in order to steal private information. Multi-use AR applications construct a "shared-state" using image sequences and GPS data. We show that these shared-states can be poisoned allowing for the reading and writing of information to and from arbitrary locations by a malicious user.

Immersive technologies are widely used in interactive media art and storytelling today. A high-quality virtual environment and avatar can provide the audience with a strong sense of presence and ownership of the virtual body, evoking empathetic outcomes from the experience. However, the majority of immersive art we encounter today shows a strong favoring of visual and auditory sensation over interactions. They provide the audience with a limited number of ways to participate in an immersive narrative, leaving them a sense of being an “outsider” that breaks the connection between the audience’s virtual avatar and the virtual environment. Starting with Nicolas Bourriaud’s relational aesthetics for art creation and taking human relations and their social context as the main conceptual and practical point of departure for experience design, I develop artistic and technical strategies for creating immersive experiences that place the audience at the center to let them activate the narrative content and cultural meanings. My approach is to provide the user novel ways to relate themselves to what they see on a screen, and to allow for direct participation and engagement. Through an increased level of interactivity, co-presence and plausibility in immersive environments, my work aims to help the audience develop new relationships with other people, with virtual entities, and with AI agents. The hybrid systems introduced in this thesis describe novel ways of creating relational experiences by techniques learned from different domains such as soft robotics, quantum computing and social science. I also demonstrate authoring interface design that can be more widely adopted by the creative community to overcome the challenges often faced in immersive and interactive narrative compositions.

Augmented Reality (AR) is an interactive technology that delivers rich and immersive computer-generated perceptual information overlaid onto the real world. AR technology continues to improve over time, potentially enabling an "always-on" AR future, where wearable AR devices are as comfortable to wear as glasses and have an all-day battery life. This concept has been termed "Pervasive Augmented Reality", in which AR is the predominant form of personal computing, instantly accessible, constantly providing information and available to assist in everyday tasks.Pervasive AR represents a shift in how, when, and where we use computers, but there are few guidelines for how to effectively create these types of systems. Current research focuses on single-purpose use cases, designed for specific tasks and environments, such as navigation or maintenance. They rarely consider AR as a mobile experience that can be taken anywhere, where computing happens continuously and persistently. This dissertation addresses that insufficiency by analyzing the design and evaluation of AR artifacts for 3 use cases: AR Recommender Systems, AR Language Learning, and AR Multitasking. We conduct our investigation in 3 parts. 1) What benefits do AR applications provide over non-AR based systems? 2) What additional inputs and signals can AR applications benefit from? and 3) How do we evolve current AR systems towards a pervasive AR future? Our work makes contributions through empirical user studies, prototype systems, and the development of new interaction techniques, revealing insights into the tools and techniques needed for developing pervasive AR, as well as the opportunities and new possibilities it enables over existing technologies.

Rapid advances in imaging have made high-quality devices such as mobile phone cameras easily accessible, opening the doors to new applications in image editing and augmentation. They may allow an interior designer to visualize how a kitchen counter will appear after remodeling, or a consumer to see whether a fabric or leather sofa looks better in a living room with color bleeding from walls of various shades, or a real estate agent to demonstrate how a room imaged under fluorescent lights at night will appear in the glow of a sunrise when a window is opened.Achieving a high degree of photorealism in such applications remains extremely challenging in computer vision and graphics. They require a comprehensive understanding of all the constituent factors of image formation — shape, material and lighting — which exhibit a wide spectrum of variations and interact in complex ways to create effects such as highlights, shadows and interreflections. Reconstruction of these intrinsic scene components or the ability to edit them is consequently an extremely ill-posed problem and especially so when only a single or a few images are available. Classical measurement-based methods need expensive, carefully calibrated setups. Prior model-based methods assume simplified physical models that break down in the face of diverse real-world appearances. Thus, a learning paradigm merits consideration, but even powerful deep learning methods suffer from a lack of generalization due to the diversity, long-range interactions and paucity of ground truth data associated with complex light transport.The key insight of this thesis is to develop physically-motivated learning, which incorporates the inductive bias of image formation to enable deep neural networks to reason about shape, material and lighting in complex scenes. The success of our approach rests on three advances. First, we develop neural differentiable rendering modules that model the full physics of image formation, including non-local light transport effects such as shadows, interreflections or refraction. Second, we devise physically-valid representations of material and light sources that are compact enough to make learning tractable, yet expressive enough to model realistic appearance such as spatially-varying reflectance or high-frequency specular highlights and light shafts through an open window. Third, we exploit domain knowledge to create large-scale photorealistic synthetic datasets which circumvent the difficulty of obtaining ground truth for spatially-varying material and complex light paths that enable physically-motivated learning to generalize well to real scenes. We demonstrate the success of our approach through results that surpass the state-of-the-art or solve longstanding open challenges in reconstruction and editing of shape, material and lighting in the presence of complex light transport in unconstrained scenes, with just a single image as input.This dissertation also democratizes research in vision and graphics through open frameworks that allow creation of high-quality virtual environments. Indeed, a key practical impact is to allow users to create realistic visual effects with only a few images captured with a mobile phone camera. Our high-quality predicted geometry, spatially-varying lighting and materials enable several augmented reality (AR) applications at an unprecedented level of photorealism — including virtual object insertion and material replacement with realistic shadows and color bleeding, transparent shape reconstruction and light source editing (such as turning off lamps or opening windows) with consistent non-local shadows, interreflections and highlights.

Augmented Reality (AR) is a medium that gives people the ability to engage with digital information in ways that deviate from more traditional HCI methods (e.g. WIMP user interfaces (Christensson, 2014)). Previous work has shown that AR has useful implications for collaboration, and with remote work experiences and teleconferencing becoming increasingly prevalent and important to people as many are working together over distance (remotely) in higher frequencies, understanding how AR might impact remote work is of particular interest. In this work, I review literature around methods for remote collaboration and using AR (and Virtual Reality (VR)) for collaboration, investigate ways that AR can support efforts to work together across distance, and study how invoking AR may create a sense of joint focus and engagement beyond what traditional remote collaboration tools afford. I explore the design process and development of an AR teleconferencing tool, SAROSvis, that allows participants to interact with one another around digital assets in AR and pass digital objects through the screen to one another while physically remote. I investigate the use cases of this tool by comparing across curated conditions and evaluating versions of this tool in user studies; I also describe the evaluation methods and user testing results of this system. This work aims to develop a research-based resource for those looking to utilize AR tools for collaboration around 3D entities or environments, with an understanding of the requirements, desires, and limitations of users and scenarios. I propose that there are meaningful ways that people can remotely collaborate in educational spaces using AR to pass (digital) objects to one another through the screen and using AR as a layer on top of the features that existing teleconferencing tools provide can make for more spatially appropriate, collaborative, engaging communication sessions while working remotely.

This study aims to evaluate the impact of using augmented reality, gamification, and serious games in computer science education. The study presents the development process of an educational mobile application, describes an experiment that was conducted and involved 117 higher education students, and analyzes the results of a 49-item paper-based questionnaire. In total, 8 research questions were explored. The results of the study revealed that several educational benefits can be yielded when integrating such applications in teaching and learning activities and actively involving students in the design and development process. In particular, the application was assessed as an effective learning tool that could enrich and improve the educational process and create interactive, inclusive, and student-centered learning environments. Its use led mostly to positive effects and experiences while maintaining the negative ones to a minimum and most students expressed positive emotions. Students were able to learn in a more enjoyable and interesting manner, and their motivation, engagement, self-efficacy, and immersion were greatly increased. Students’ innate need for autonomy, competence, and relatedness was satisfactorily met and both their intrinsic and extrinsic learning motivations were triggered. They felt a sense of belonging and cultivated their social skills. The potential of the application to improve students’ knowledge acquisition and academic achievements was also observed. The application also enabled students to improve their computational thinking and critical thinking skills. Therefore, the potential of combining augmented reality, gamification, and serious games to enhance students’ cognitive and social–emotional development was highlighted.

The internet-of-things (IoT) refers to the growing number of embedded interconnected devices within everyday ubiquitous objects and environments, especially their networks, edge controllers, data gathering and management, sharing, and contextual analysis capabilities. However, the IoT suffers from inherent limitations in terms of human-computer interaction. In this landscape, there is a need for interfaces that have the potential to translate the IoT more solidly into the foreground of everyday smart environments, where its users are multimodal, multifaceted, and where new forms of presentation, adaptation, and immersion are essential. This work highlights the synergetic opportunities for both IoT and XR to converge toward hybrid XR objects with strong real-world connectivity, and IoT objects with rich XR interfaces. The paper contributes i) an understanding of this multi-disciplinary domain XR-IoT (XRI); ii) a theoretical perspective on how to design XRI agents based on the literature; iii) a system design architectural framework for XRI smart environment development; and iv) an early discussion of this process. It is hoped that this research enables future researchers in both communities to better understand and deploy hybrid smart XRI environments.

The internet-of-things (IoT) refers to the growing field of interconnected pervasive computing devices and the networking that supports smart, embedded applications. The IoT has multiple human-computer interaction challenges due to its many formats and interlinked components, and central to these is the need to provide sensory information and situational context pertaining to users in a more human-friendly, easily understandable format. This work addresses this by applying mixed reality toward expressing the underlying behaviors and states internal to IoT devices and IoT-enabled objects. It extends the authors’ previous research on IoT Avatars (mixed reality character representations of physical IoT devices), presenting a new head-mounted display framework and interconnection architecture. This contributes i) an exploration of mixed reality for smart spaces, ii) an approach toward expressive avatar behaviors using fuzzy inference, and iii) an early functional prototype of a hybrid physical and mixed reality IoT-enabled object. This approach is a step toward new information presentation, interaction, and engagement capabilities for smart devices and environments.

During crisis situations, emergency operators require fast information access to achieve situation awareness and make the best possible decisions. Augmented reality could be used to visualize the wealth of user-generated content available on social media and enable context-adaptive functions for emergency operators. Although emergency operators agree that social media analytics will be important for their future work, it poses a challenge to filter and visualize large amounts of social media data. We conducted a goal-directed task analysis to identify the situation awareness requirements of emergency operators. By collecting tweets during two storms in Germany we evaluated the usefulness of Twitter data for achieving situation awareness and conducted interviews with emergency operators to derive filter strategies for social media data. We synthesized the results by discussing how the unique interface of augmented reality can be used to integrate social media data into emergency control rooms for situation awareness support.

This study aims to establish Middle East users' perspectives on the major factors that impact their\ud decision to adopt Augmented Reality AR smart glasses (ARSG). Thus, an online questionnaire was\ud designed and sent directly to the respondents, and 584 valid data points were collected from individuals living in the Middle East. The data were analyzed using Pearson correlations and Exploratory\ud Factor Analysis (EFA) techniques using SPSS. Eleven hypotheses were tested using Multiple Regression analysis, where seven independent variables out of eleven were confirmed to have a significant impact on the perceived adoption of ARSG. The results indicate that four of the independent\ud variables including Pre-Market Knowledge, Image, Own privacy and Technology innovativeness\ud show the significant impact on ARSG adoption at the 1% significant level. In addition, the results\ud indicate that three of the social and technological factors include Perceived Ease of use, Perceived\ud usefulness and Other's privacy show the significant effect on ARSG adoption at the 5% significant\ud level. Among the 7 social and technological factors, the results suggest that technology innovation\ud expresses the strongest effect on ARSG adoption with the highest coefficient value of 0.413 (b =\ud 0.413, t = 12.881, ρ < 0.01). Moreover, user intention is significantly impacted by gender and place\ud of living but not by education or age. The research also provides pre-market insights on users' personal types that represent who will most likely adopt the new smart glasses and that differentiate\ud them based on their priorities. To the best of our knowledge, this is among the first works to investigate technology acceptance drivers of AR smart glasses in the Middle East.

Mutual occlusion is an essential feature for augmented reality (AR) displays for allowing the virtual content to be clearly perceived under an excessively illuminated environment. Although a few works have been done to facilitate the performance of occlusion-capable optical see-through augmented reality (OC-OST-AR) displays, the realization of mutual occlusion in a wide field-of-view (FOV) is still challenging. Divergent from typical hard-edge occlusion and soft edge-occlusion designs, we propose the paired-ellipsoidal-mirror (PEM) structure. The proposed system is allowed to support either hard-edge occlusion or enhanced soft-edge occlusion in a wide FOV by optionally fixing a spatial light modulator (SLM) before the entrance pupil or at an inner focal plane. The numerical aperture (NA) of the system is efficiently increased by the combination of paired ellipsoidal mirror imaging and aperture stop restriction. With proof-of-concept prototypes built, virtual display in a FOV of H160°×V74° and mutual occlusion in a FOV of H122°×V74° are demonstrated with a basic design, respectively. Furthermore, a mixed FOV of H95.3°×V52.9° is demonstrated by an optimized design with vertical parallax reduction and virtual display improvement.

The amount of arable land is limited, yet the demand for agricultural food products is increasing. This issue has led to the notion of precision farming, where smart city-based technologies (e.g., Internet of Things, digital twins, artificial intelligence) are employed in combination to cater for increased production with fewer resources. Widely used in manufacturing, augmented reality has demonstrated impactful solutions for information communication, remote monitoring and increased interaction. Yet, the technology has only recently begun to find a footing alongside precision farming solutions, despite the many benefits possible to farmers through augmenting the physical world with digital objects. Therefore, this article reflects on literature discussing current applied solutions within agriculture, where augmented realty has demonstrated a significant impact for monitoring and production. The findings discuss that augmented reality must be coupled with other technologies (e.g., simultaneous localization and mapping algorithms, global positioning systems, and sensors), specifically 9 are identified across 2 application domains (livestock and crop farming) to be beneficial. Attention is also provided on how augmented reality should be employed within agriculture, where related-work examples are drawn from in order to discuss suitable hardware approaches and constraints (e.g., mobility).

Cultural-heritage research has created a vast amount of information regarding heritage objects and sites. At the same time, recent efforts on the digitization of cultural heritage have provided novel means of documenting tangible cultural resources including digital images, videos, audio testimonies, and 3D reconstructions. Furthermore, ethnographic research is nowadays combined with advanced capturing technologies such as motion capture to record the intangible dimensions of heritage as these are manifested through human expression in dance, heritage crafts, etc. This amount of information is now available and should be used to create novel forms of experiential access to cultural heritage powered by the web and mobile technologies mixed with novel interaction paradigms such as virtual and augmented reality. In this article, a platform is presented that facilitates a cloud-based web application to support location-based narratives on cultural-heritage resources provided through map-based or story-based representation approaches. At the same time, the platform through the power of modern mobile devices can provide these experiences on the move using location-based and image recognition-based augmented reality to facilitate multiple usage contexts. The platform was implemented to support the goal of the project CuRe, in the context of the “Greece-Germany” bilateral collaboration action.

The rapid development of virtual and augmented reality technologies is currently taking place in almost all spheres of activity. Elements of virtual and augmented reality are used in such areas as education, medicine, transport, gaming, tourism and others. The active spread of these technologies causes the emergence of special competencies in the IT labor market and, as a result, the formation of new professions.Many Russian universities are training students in IT training areas. Specialization in the development of computer games and virtual reality applications has begun recently. The provision of practical classes is accompanied by specific tasks, which gives students the opportunity to improve the use of software and technical devices.The relevance of the research is determined by the current demand for the use of the latest technologies by IT developers in the field of creating computer games. Today, technologies that provide a player’s immersion in virtual reality are becoming more and more popular. One of these technologies is a suit with wearable sensors that track a person’s position in space in real time. However, there are quite a few real described projects in the literature and on the Internet. This study examines the process of developing a task for creating a game application using virtual reality technology: a suit with wearable sensors for teaching students.Materials and methods of research. Timely identification of the needs of the IT market in personnel training allows educational organizations to form new training programs of different levels of training. This approach makes it possible to target the educational and methodological materials being developed to use the latest achievements in the development of the field under study.Using a systematic approach, the study characterizes virtual reality suits and sensors for monitoring the position in the user’s space. Thus, the goal of the task was to ensure the immersiveness and convenience of interaction between the player and the game environment.Based on materials on software, position sensors in space, the approach of pedagogical design was applied and the procedure was formed for a practical task, reflecting the relevant competencies.Results. The study was conducted on the basis in the framework of laboratory and practical work of students, as well as at a real enterprise. Training in the new profile of the direction of training “Applied informatics” is fully equipped with all the latest technologies in this field. As a result of the work, the content of the practical task was developed.Real development of virtual and augmented reality applications is conducted jointly with students. Almost all projects used a suit with body sensors.Conclusion. Our study examines in detail the process of developing an application using a suit with wearable sensors for further training of students. Based on the results, work can be carried out on real projects for any field. Based on the research materials, it is planned to issue a textbook for students with the profile of developing computer games and virtual / augmented reality applications.

Augmented reality (AR) and virtual reality (VR) are burgeoning technologies that have the potential to greatly enhance patient care. Visualizing patient-specific three-dimensional (3D) imaging data in these enhanced virtual environments may improve surgeons’ understanding of anatomy and surgical pathology, thereby allowing for improved surgical planning, superior intra-operative guidance, and ultimately improved patient care. It is important that radiologists are familiar with these technologies, especially since the number of institutions utilizing VR and AR is increasing. This article gives an overview of AR and VR and describes the workflow required to create anatomical 3D models for use in AR using the Microsoft HoloLens device. Case examples in urologic oncology (prostate cancer and renal cancer) are provided which depict how AR has been used to guide surgery at our institution.