Search

ASD is a neurodevelopmental condition characterised by impairments in social communication and interaction and restricted or repetitive patterns of behaviour. Early intervention is recommended to produce the best outcomes for children with ASD. Clinicians therefore use specific therapies that most effectively improve outcomes for children. An intervention that is widely used by teachers and therapists is role play. Role play is a form of pretend play which is widely used by educators and therapists, to improve a variety of skills in children with ASD, including social and communication skills and Theory of Mind (ToM). ToM is the ability to attribute mental state to others, or in other words, the ability to 'put yourself in someone else's shoes'. In recent years, digital interventions to support learning for children with ASD which use interactive game-based technology have been increasing. Augmented reality (AR) is one such technology that is beginning to show promising results. AR is an advanced technology, commonly using visual cues, that expands our world by superimposing digital content on top of reality. Several unique features of AR have been shown to support learning for children with Autism Spectrum Disorders (ASD). This study explores the design and creation of an innovative augmented reality pretend play intervention for children with autism spectrum disorders (ASD), through a user centred design process (UCD), also known as human centred design. A conceptual system framework was presented for researchers to use to create digital pretend play interventions for children with ASD. The conceptual system framework consisted of user requirements, instructional guidelines, a system template as well as recommendations of suitable technologies. The conceptual system framework was used to design a prototype intervention for children with ASD, to improve Theory of Mind through an AR based role play (aka socio-dramatic play) activity. The intervention uses the unique affordances of AR, which allow the user to control a digital character using their own facial expressions to facilitate role play activities. The intervention progressed through a rapid prototyping process where user feedback was gained through UCD methods, including focus group and usability testing. The user centred design approach was then adopted to ascertain and validate the design of the application and the user requirements through two usability studies. The final design is an iPad based AR application using facial expression tracking to interact with the activity. The digital intervention is designed to be used by a child and a guiding adult as a tool to extend and encourage role play. The digital intervention activity is based on familiar characters set in the context of a story book from the Oxford Reading Tree scheme, however, any character-based story book or media could be used. The child uses this to "pretend to be" a character in the story and accurately present their character's emotions at different points in the narrative. The impact of the application on Theory of Mind and overall engagement was evaluated through two sets of case studies, with a total of eight children in two different schools in the North West of the UK. The results establish that an AR face driven interface supports the development of Theory of Mind in children with ASD. The study demonstrated strong evidence for improvement of Theory of Mind for the participants of the study. Engagement in the activity across all cases was very high. This work could lead to interesting future studies related to AR technologies for children with ASD as increased engagement could lead to improved learning and outcomes. The two contributions of this study are, firstly, the user requirements and conceptual system framework generated through a UCD approach which could be used by researchers in the design of digital pretend play interventions for children with ASD. Secondly, the creation of an AR intervention which shows strong support for the development of Theory of Mind for children with ASD. Future research and larger scale studies are recommended to validate the results across a wider group of children with ASD and to determine which subsets of children with ASD would benefit the most from this intervention. The intervention should be studied further for its potential to improve social conversation in children with ASD.

Over the years, a set of story-telling conventions has been established for Television (TV), between content creators and their audiences. These conventions typically involve the creation, and subsequent consumption, of 2-dimensional (2-D) imagery accompanied by sound, requiring limited, if any, interaction between the viewers and content. As consumer Augmented Reality (AR) devices become a commodity, researchers and broadcasters are investigating the ways in which AR can be used to enhance/transform a conventional TV viewing experience - a concept referred to as Augmented Reality Television (ARTV). However, the addition of AR to TV has the potential to disrupt the conventional TV viewing assumptions, since AR's key characteristics are arguably its ability to deliver interactive 3-dimensional (3-D) imagery that is registered to the physical space; a type of content that TV viewers are likely not used to consuming. This raises the question of how to create ARTV content? This thesis conceptualises ARTV as an emerging mass medium, where content creators use both AR and TV to tell their stories to their audiences. We start by providing a few preliminary working definitions (e.g., TV, AR, and ARTV), then systematically review prior research and identify six major themes and six prominent design dimensions (namely, abstraction, interaction, time, display, context, and editorial control). These design dimensions are then used as a basis to create an ARTV design space, from which we derive a cheat sheet to support practitioners. We subsequently evaluate the intelligibility of this cheat sheet, and its impact on conceptualising novel ARTV experiences, in a user study with n = 10 participants. Our findings indicate that the cheat sheet can be useful in conceptualising novel ARTV scenarios, and prompt a series of refinements to the design space and associated cheat sheet. Given the dominance of ARTV to enhance conventional TV viewing experiences that take place in the living room, we further develop this aspect with its own conceptual framework and classification. To demonstrate the applicability of our design space and conceptual framework, we explore two specific types of living room based ARTV experiences in two user studies. In the first, we add holograms to an existing nature documentary and invite n = 12 participants to watch a six-minute ARTV content - where TV content is presented on a physical TV display, and AR content is displayed on a Microsoft Hololens head-mounted display - in a user testing facility that is designed to look and feel like a conventional living room. Our findings suggest that the viewers are likely to find such an experience engaging. However, some may feel a degree of anxiety and a fear of missing parts of the content and underlying narrative. In the second study (n = 10), we use points on the display dimension of our design space to vary presentation of a lifelike programme-related hologram as part of a nature documentary show. We explore six points on the display dimension in the context of an ARTV experience where both the TV and the living room are virtual. We investigate the impact of the hologram's movement behaviour on the viewers. Our findings highlight the role of personal preferences and the perceived role of the hologram in the underlying narrative and TV content. Overall, the thesis highlights a need to carefully consider the impact of additional augmentations on the viewers. Furthermore, viewers should be given the ability to customise their ARTV experiences, based on personal preferences. Our contributions are the following: 1) a systematic literature review, highlighting six common themes of research in ARTV and six commonly used design dimensions; 2) creation and evaluation of a design space for ARTV and its subsequent refinement based on the evidence gathered from a user study; 3) a conceptual framework for ARTV experiences in the living room, identifying nine types of ARTV and highlighting areas for future research; 4) two user-studies investigating two types of ARTV, highlighting a need to accommodate personal preferences for ARTV content.

This portfolio of compositions uses novel biometrics from gestural interfaces called Myo armbands to compose five pieces of electroacoustic music within a continuum of spatial environments, ranging from the real to the virtual. The portfolio achieves this by processing biometric data from the Myo armbands within developed software (using Max 8), applying machine learning to such data (via Wekinator) and mapping predictive outputs to audiovisual materials across the spatial continuum. The results of this approach show how unique compositional affordances can be created when using biometrics within different spatial environments and how implementing machine learning methods can help to create such opportunities. The outcomes are presented via five composed portfolio works, three published papers within the scientific community, and numerous music performances navigating the overlapping areas of music composition and computer science. By using novel biometrics to compose music within a continuum of space, this interdisciplinary research topic contributes original knowledge to both the fields of music composition and computer science (human-computer interaction).

Dementia affects more than 850,000 people in the UK, and this figure continues to rise year on year. People living with dementia often have difficulties completing activities of daily living (ADLs), leading to a reliance on family or professional carers. However, assistive technology, such as task prompting tools, can support people living with dementia to maintain their independence and live at home for longer. Augmented Reality (AR) is an increasingly prevalent technology and has been used as a task prompting tool in industrial settings to support complex maintenance and assembly tasks. The use of AR for task assistance has also been identified as a promising area for development in Domestic AR. Despite the use of AR as a task prompting tool in industrial settings and the potential of AR in domestic settings, relatively little is known about the efficacy of augmentations for ADLs. The work in this thesis aims to provide an initial exploration into the use of AR as a task prompting tool to support people living with dementia to complete ADLs at home. Multiple stakeholders, including health professionals, a general adult population, older adults without cognitive impairment, and people living with dementia and their family carers, have been included to develop a holistic understanding of the use of AR in a domestic setting, based on four studies carried out for this project. The first study consisted of in-person interviews with professionals with experience of working with people living with dementia. The second study was a lab experiment with older adults to compare four AR visual prompting techniques to prompt five basic actions found in many ADLs. The third study involved the co-design of AR prompts in a kitchen context, and their evaluation with a general adult population using an online survey. The final study consisted of online interviews with people living with dementia and their family carers to explore the results of the previous three studies and how AR could be beneficial from the point of view of people with lived experience of dementia. The overall findings show that AR as a tool to support task prompting of domestic tasks was received positively by the participants of these studies. A combination of text, audio, and a ghosthand image demonstrating the action to carry out could be most beneficial for people with dementia, but AR prompts should be easily customisable to cater for different abilities, preferences, and personalities. Furthermore, early introduction of AR will be key for uptake when the technology has been developed further. The potential of domestic AR to improve the lives of people affected by dementia and those that support them with ADLs is considered as motivation for future work in this promising research area.

This thesis presents an analysis of the implementation of Spatial Augmented Reality (SAR) to support collaborative design. SAR is a class of Augmented Reality (AR), where the AR effect is independent from the viewer's perspective. This property should make SAR uniquely well suited to support collaborative design sessions as it allows multiple participants to simultaneously view the same AR enhanced models concurrently. However, the adoption of SAR in the field of design remains limited. Thus, there is a knowledge gap that surrounds SAR, due to the novelty of the technology, and how its implementation can be configured to better support design. The aim of this research is therefore to: "Investigate SAR Systems' Characteristics and Features and how these Affect the Efficacy of Co-Design Sessions". A review of the literature surrounding collaborative design is provided to highlight evidence that enhanced collaboration during design sessions results in improved outcomes. Thereafter AR, and SAR in particular, are analysed to provide a basis for the continued investigation of SAR as a valid tool to support co-design sessions. In addition to the review, the state-of-the-art in implementing SAR in design and collaborative design is presented and discussed to frame the subsequent research conducted as part of this PhD. Moreover, literature is provided covering the metrics that will be used to analyse SAR supported co-design sessions. The methodology implemented throughout this thesis was one of empirical research conducted through a number of experimental studies aimed at evaluating the impact of SAR on co-design sessions. The experimental nature of this research thus required the development of an SAR platform to be used during experimental testing. The platform was developed iteratively and improved on the basis of feedback received from experiments. These experiments focused on three main areas: 1. The impact of SAR on co-design sessions as a whole. These experiments aid in understanding the impact that SAR has on the overall design session, focusing primarily on the design outcomes. 2. The analysis of specific characteristics and features of SAR and their impact on design outcomes. Experiments used to analyse the impact of specific elements that are innate to SAR to evaluate their individual impact on SAR supported design sessions. 3.Experiments and data collection aimed at gathering industry feedback. Used to understand the industry perspective to allow for the optimization of SAR to cater to the needs of industry and ultimately allow for its wider adoption. The results highlighted the viability of SAR as a tool to support collaborative design sessions. However, the studies also identified a number of areas for improvement in the implementation of SAR. The thesis concludes with a summary of thirty-two findings, condensing them into two lists of recommendations. One of these lists for technical improvements and one for features and affordances, aimed at assisting future researchers wishing to develop improved SAR platforms for co-design.

To address some of the challenges associated with aging bridge infrastructure, this dissertation explores the development and evaluation of a novel tool for bridge inspections leveraging Augmented Reality (AR) and computer vision (CV) technologies to facilitate measurements. Named the Wearable Inspection Report Management System (WIRMS), the system supports various data entry methods and an adaptable automation workflow for defect measurements, showcasing AR's potential to improve bridge inspection efficiency and accuracy. Within this context, the work's main research goal is to understand the difference in performance between traditional field data collection methods (i.e. pen and paper) and automated methods like spoken data entry and CV-based structural defect measurements. In case of CV assistance, emphasis was placed on human-computer interaction (HCI) to understand whether partial, collaborative automation could address some of the limitations of fully automated inspection methods. The project began with comprehensive data collection through interviews, surveys, and observations at bridge sites, which informed the creation of a Virtual Reality (VR) prototype. An initial user study tested the feasibility of using voice commands for data entry in the AR environment but found it impractical. A second user study focused on optimizing interaction methods for virtual concrete crack measurements by testing different degrees of automated CV assistance. As part of this effort, major technical contributions were made to back-end technologies and CV algorithms to improve human-machine collaboration and ensure the accuracy of measurements. Results were mixed, with larger degrees of automation resulting in significant reductions in inspection time and perceived workload, but also significant increases in the amount of measurement error. The latter result is strongly associated with a lack of field robustness of CV methods, which can under-perform if conditions are not ideal. In general, hybrid techniques which allow the user to correct CV results were seen as the most favorable. Field validations with bridge inspectors showed promising potential for practical field implementation, though further refinement is needed for broader deployment. Overall, the research establishes a viable path for making AR a central component to future inspection practices, including digital data collection, automation, data analytics, and other technologies currently in development.

Augmented Reality (AR) devices are set apart from other mobile devices by the immersive experience they offer. Specifically, head-mounted AR devices can accurately sense and understand their environment through an increasingly powerful array of sensors such as cameras, depth sensors, eye gaze trackers, microphones, and inertial sensors. The ability of these devices to collect this information presents both challenges and opportunities to improve existing security and privacy techniques in this domain. Specifically, eye gaze tracking is a ready-made capability to analyze user intent, emotions, and vulnerability, and as an input mechanism. However, modern AR devices lack systems to address their unique security and privacy issues. Problems such as lacking local pairing mechanisms usable while immersed in AR environments, bystander privacy protections, and the increased vulnerability to shoulder surfing while wearing AR devices all lack viable solutions. In this dissertation, I explore how readily available eye gaze sensor data can be used to improve existing methods for assuring information security and protecting the privacy of those near the device. My research has presented three new systems, BystandAR, ShouldAR, and GazePair that each leverage user eye gaze to improve security and privacy expectations in or with Augmented Reality. As these devices grow in power and number, such solutions are necessary to prevent perception failures that hindered earlier devices. The work in this dissertation is presented in the hope that these solutions can improve and expedite the adoption of these powerful and useful devices.

This dissertation, utilizing a mixed-methods approach within a design-based research (DBR) framework, investigated the impact of augmented reality (AR) technology on the self-efficacy of preservice elementary teachers (PSTs) in integrated STEM teaching and technology. Incorporating both a pilot study and a subsequent dissertation study, the research specifically focused on the effects of AR GeoGebra. The key results that emerged from the pilot and dissertation studies highlight AR's potential to significantly enhance PSTs' STEM teaching self-efficacy. In both studies, the treatment group that engaged with AR GeoGebra showed substantial improvements in integrated STEM teaching self-efficacy. Additionally, in both studies, participants from all groups exhibited significantly improved technological self-efficacy after using GeoGebra, regardless of AR exposure. Qualitative data underscored AR's role in promoting hands-on learning, fostering creativity in STEM education, and bolstering positive attitudes towards STEM integration. These findings emphasize the transformative potential of AR in STEM teacher preparation and its capability to address the needs of preservice teachers in technology-enhanced educational environments.

Full-body powered exoskeletons for industrial workers have the potential to reduce the incidence of work-related musculoskeletal disorders while increasing strength beyond human capabilities. However, operating current full-body powered exoskeletons imposes different loading, motion, and balance requirements on users compared to unaided task performance, potentially resulting in additional mental workload on the user which may reduce situation awareness (SA) and increase risk of collision with pedestrians, negating the health and safety benefits of exoskeletons. Exoskeletons could be equipped with visual aids to improve SA, like rearview cameras or radar displays. However, research on design and evaluation of such displays for exoskeleton users are absent in the literature. This empirical study compared several augmented reality (AR) head-up displays (HUDs) in providing SA to minimize pedestrian collisions while completing common warehouse tasks. Specifically, the study consisted of an experimental factor of display abstraction including four levels, from low to high abstraction: rearview camera, overhead radar, ring radar, and no visual aid (as control). The second factor was elevation angle that was analyzed with the overhead and ring radar displays at 15°, 45°, and 90°. A 1x4 repeated measures ANOVA on all four display abstraction levels at 90° revealed that every display condition performed better than the no visual aid condition, the Bonferroni post-hoc test revealed that overhead and ring radars (medium and high abstraction respectively) received higher usability ratings than the rearview camera (low abstraction). A 2x3 repeated measures ANOVA on the two radar displays at all three display angles found that the overhead radar yielded better transport time and situation awareness ratings than the ring radar. Further, the two-way ANOVA found that 45° angles yielded the best transport collision times. Thus, AR displays presents promise in augment SA to minimize collision risk to collision and injury in warehouse settings.

As next-generation space exploration missions require increased autonomy from astronaut crews, real-time diagnostics of astronaut health and performance are essential for mission operations, especially for determining Extravehicular Activity (EVA) readiness. Exposure to microgravity leads to changes in astronaut physiology such as loss of bone density and muscle mass, cardiovascular deconditioning, and the reorganization of sensorimotor integration that require distinct adaptation timelines. As astronauts transition from microgravity to gravity-rich environments (e.g., the Moon or Mars), sensorimotor impairment may affect their ability to perform mission-critical tasks such as landing procedures, capsule egress, and early EVA. To ensure mission success and astronaut safety, it is essential to identify sensorimotor impairment during missions. At present, no flight-ready tools are available for astronauts to assess their sensorimotor impairment throughout a mission, especially tools for which expert assistance is not needed. Current Earth-based assessments require extensive resources to administer and skilled clinicians to score and interpret the data. The development of lightweight, space-conscious assessments for astronauts transitioning between gravity environments is crucial to the success of future exploration missions. An Augmented Reality (AR) standalone system may be a viable solution by allowing holographic visual cueing to replace physical objects used in traditional assessments. AR headsets are easily deployable, enable the user to see their physical environment for safety in confined spaces, provide flexibility for new software integration, and are multifunctional for other mission tasks such as procedural guidance. The research focused on the development, usability testing, and evaluation of the Augmented Reality Operations Readiness Assessment (AURORA) to assess neurovestibular and sensorimotor performance with holographic visual cueing and integrated inertial measurement units. The human-in-the-loop usability testing demonstrated positive perceptions of usability across two diverse age groups spanning current and future astronaut age demographics. Although usability concerns were identified with a subset of users, future design recommendations, such as proximity lighting for improved depth perception, are provided to promote self-administration for all users. While performance for the sensorimotor and vestibular assessments differed when compared to Earth-based standards and physical assessments, the research demonstrated the potential of AR as an assessment tool with the benefit of embedded sensors and onboard computing capabilities. The AR tool effectively quantified changes in hand-eye coordination performance as measured by accuracy, precision, error rate, throughput, and movement time with varying task difficulty. Users were able to successfully complete the tasks, yielding meaningful performance measures. When compared to the physical environment, AR led to altered user strategies, predominantly marked by increased task time, reduced head and torso movements, and increased head pitch amplitude aimed at enhancing hologram visibility due to the restricted field of view of the headset. User performance could have also been attributed to the novel environment for most users. AR-induced strategies resembled compensatory responses observed in astronauts after spaceflight and in vestibular patients, suggesting that AR would not exacerbate symptoms of motion sickness. These findings address gaps outlined in NASA’s Human Research Roadmaps and offer guidance for AR design within NASA’s Human Integration Design Handbook. The research advances AR's potential as a standalone assessment tool in evaluating astronaut performance throughout missions while considering spaceflight constraints. The implication of this research extends into applications for both the aerospace industry and the medical field as it relates to aging populations and individuals with balance and sensorimotor disorders.

This paper examines the theory and design considerations of effective learning applications built with Augmented Reality (AR) to better prepare Basic Combat training candidates for the tasks and stressors associated with training. The research questions focused on the evaluation of learning outcomes, emotional experiences, and transfer results compared between traditional multimedia material and a multimodal AR application designed and developed for this study. A primary interest is to discover if informal learning can prime basic training candidates for the rigors of basic training and increase their successful completion. A mixed-methods, between-subject research design was executed as a pilot study (n = 10). Two groups were compared. The control group (n = 5) represented the standard educational practices that use-multimedia slides and the experimental group (n = 5) introduced a novel-AR App. A variety of instruments were used to measure declarative knowledge, conceptual knowledge, procedural task knowledge, and emotional states, followed by a time delayed transfer test on the same outcomes. Results that were evaluated using non-parametric statistics show there are no statistical differences in actual learning or transfer between the two groups but that there are strong trends in perceived learning, perceived stress, homesickness, rankings regarding the utility of AR, and subjective evaluation (immersion, engagement, and perceived learning). Interesting correlations were found to be different between the AR and the control conditions that may inform the use of technology in training applications and design. Qualitative data was gathered using interviews, think-aloud protocols, and field notes. This quantitative data combined iv with qualitative evidence supports further study with an increased sample size to better evaluate immersive learning application design.

In terms of changing production in the transformation to the Industry 4.0 landscape and increasing demand for customized products, small and medium enterprises (SMEs) utilizing industrial robots encounter a shortage of skilled labor. Open questions in human-robot interaction encompass the need for natural and effective interfaces, path planning within intricate environments, and the design and execution of human-robot collaborations. This dissertation aims to tackle these challenges by leveraging Augmented Reality (AR) to enhance robot operations in manufacturing scenarios, with a focus on optimizing the efficiency of the Human-Robot Interaction (HRI) system, improving the bidirectional trust between humans and robots, and fostering effective collaboration. The first topic in this dissertation is the development of HA$R^{2}$bot, an AR robot programming interface with online and offline methods for path planning in complex environments, verified for intuitiveness, efficiency, and usability through a user study. The second focal point pertains to the registration issues encountered in AR systems, encompassing error identification and propagation, which are mitigated through the introduction of a global view calibration method. The third core area addresses the formidable challenges in the implementation of complex, long-term human-robot collaborative assembly tasks. This is achieved through the introduction of a 5-layer Digital Twin model, an Augmented Reality interface, and a task planner framework, all working in tandem to enhance the precision and efficiency of robotic assembly processes.

This research explores how mobile augmented reality experiences are perceived and interacted with by using a quantitative eye-tracking study to record user eye movement behavior. Visual communication is vital in today’s increasingly visual society and current technology gives evidence of much promise in adding additional engagement, information, and interactivity through augmented reality. Though smartphone technology allows all smartphone users to access and experience browser-based augmented reality, there is little research on design aspects from a user experience perspective, such as the influence of layered depths in an augmented scene. The purpose of this study was to explore the variable of augmented reality depths and their impact on user engagement and interest within the framework of perception theory. A quantitative research method of eye tracking was used to measure fixations, and a user experience Likert-style question was used to assess the level of interest between single-layer and multiple-layer augmented reality experiences. The results found that total gaze time was increased for one of the multilayered stimuli, and participants ranked the multilayered experiences as more interesting than the single layer. This study has theoretical implications within the visual communication perception theory and methodological implications for eye tracking and user experience. This study also has practical implications for content creators that seek to enhance visual communication engagement and interest through augmented reality and suggestions are offered for future research.

Augmented reality (AR) is seeing a rapid expansion into several domains due to the proliferation of more accessible and powerful hardware. While augmented reality user interfaces (AR UIs) allow the presentation of information atop the real world, this extra visual data potentially comes at a cost of increasing the visual clutter of the users' field of view, which can increase visual search time, error rates, and have an overall negative effect on performance. Visual clutter has been studied for existing display technologies, but there are no established measures of visual clutter for AR UIs which precludes the study of the effects of clutter on performance in AR UIs. The first objective of this research is to determine the applicability of extant image analysis measures of feature congestion, edge density, and sub-band entropy for measuring visual clutter in the head-worn optical see-through AR space and establish a relationship between image analysis measures of clutter and visual search time. These image analysis measures are specifically chosen to quantify clutter, as they can be applied to complex and naturalistic scenes, as is common to experience while using an optical see-through AR UI. The second objective is to examine the effects of AR UIs comprised of multiple apparent depths on user performance through the metric of visual search time. The third objective is to determine the effects of other AR UI properties such as target clutter, target eccentricity, target apparent depth and target total distance on performance as measured through visual search time. These results will then be used to develop a visual clutter score, which will rate different AR UIs against each other. Image analysis measures for clutter of feature congestion, edge density, and sub-band entropy of clutter were correlated to visual search time when they were taken for the overall AR UI and when they were taken for a target object that a participant was searching for. In the case of an AR UI comprised of both projected and AR parts, image analysis measures were not correlated to visual search time for the constituent AR UI parts (projected or AR) but were still correlated to the overall AR UI clutter. Target eccentricity also had an effect on visual search time, while target apparent depth and target total distance from center did not. Target type and AR object percentage also had an effect on visual search time. These results were synthesized into a general model known as the "AR UI Visual Clutter Score Algorithm" using a multiple regression. This model can be used to compare different AR UIs to each other in order to identify the AR UI that is projected to have lower target visual search times.

As the frontiers and futures of work evolve, humans and machines will begin to share a more cooperative working space where collaboration occurs freely amongst the constituent members. To this end, it is then necessary to determine how information should flow amongst team members to allow for the efficient sharing and accurate interpretation of information between humans and machines. Shared situation awareness (SSA), the degree to which individuals can access and interpret information from sources other than themselves, is a useful framework from which to build design guidelines for the aforementioned information exchange. In this work, we present initial Augmented/virtual reality (AR/VR) design principles for shared situation awareness that can help designers both (1) design efficacious interfaces based on these fundamental principles, and (2) evaluate the effectiveness of candidate interface designs based on measurement tools we created via a scoping literature review. This work achieves these goals with focused studies that 1) show the importance of SSA in augmented reality-supported tasks, 2) describe design guidelines and measurement tools necessary to support SSA, and 3) validate the guidelines and measurement tools with a targeted user study that employs an SSA-derived AR interface to confirm the guidelines distilled from the literature review.

Many people desire to learn a new skill or craft, but having access to a mentor available to provide one on one feedback can be expensive or inaccessible to many. Learning a new skill can bring meaning and added value to a person’s life. Glasses explores bridging the physical and digital world through augmented reality tools that assist in the hand making of products and the learning of a new craft. Though Glasses specifically focuses on glass blowing, the concept could just as easily be applied to any form of craftsmanship such as woodworking, metalsmithing or welding. The product helps inform users of the best way to move in 3D space to create their desired form. Additionally, they can view a model projected onto the material they’re working to better understand and reference the form they’re trying to create. It is easy to use, and relatively hands free featuring a simple, customizable physical interface. Having direct feedback from an expert is often the best way to learn a new skill, Glasses is the next closest thing for those without one. Glasses is a full product line, featuring a physical set of safety glasses and the software to go along with them.

Maintaining good oral hygiene is crucial for a healthy life, and properly tooth brushing stands out as a fundamental defense against dental issues. Unfortunately, many people do not brush their teeth properly and there is very limited technology available to assist them in compliance with the standard toothbrushing procedure. In the realm of technological innovation, augmented reality stands out as a powerful tool with boundless possibilities. Leveraging the potential of this transformative technology, we aim to utilize mirrors as a unique medium to design a captivating 3D projected model of teeth. This innovative approach aims to inspire and motivate users to elevate their brushing habits, ultimately leading to optimal oral health.

Extended reality, or "XR", is the adopted umbrella term that is heavily gaining traction to collectively describe Virtual reality (VR), Augmented reality (AR), and Mixed reality (MR) technologies. Together, these technologies extend the reality that we experience either by creating a fully immersive experience like in VR or by blending in the virtual and "real" worlds like in AR and MR. The sustained success of XR in the workplace largely hinges on its ability to facilitate efficient user interactions. Similar to interacting with objects in the real world, users in XR typically interact with virtual integrants like objects, menus, windows, and information that convolve together to form the overall experience. Most of these interactions involve near-field object manipulation for which users are generally provisioned with visual representations of themselves also called self-avatars. Representations that involve only the distal entity are called end-effector representations and they shape how users perceive XR experiences. Through a series of investigations, this dissertation evaluates the effects of virtual end effector representations on near-field object retrieval interactions in XR settings. Through studies conducted in virtual, augmented, and mixed reality, implications about the virtual representation of end-effectors are discussed, and inferences are made for the future of near-field interaction in XR to draw upon from. This body of research aids technologists and designers by providing them with details that help in appropriately tailoring the right end effector representation to improve near-field interactions, thereby collectively establishing knowledge that epitomizes the future of interactions in XR.

This thesis project is an attempt to create an integration of augmented reality (AR) technology and tabletop board games, with a focus on tackling the challenge of maintaining engagement and capturing the attention of modern audiences who are often drawn to more interactive forms of entertainment like movies and video games. Specifically, the project explores the concept of augmenting older tabletop games with a combination of futuristic digital graphics and overlays with original physical game elements. The project sees and recognizes the problem, which is that the disparity between the level of engagement offered by traditional tabletop games and the immersive experiences provided by modern digital entertainment is quite large. It identifies the problem of older tabletop games lacking the ability to sustain audience interest, particularly in comparison to their digital counterparts. To bridge this gap, the thesis proposes the use of AR technology, which offers a promising avenue to enhance the gameplay experience by introducing visually captivating and interactive elements. The project uses the 1995 board game Settlers of Catan as a building block and basic formula for one example of how AR can be used to bridge the engagement gap (Catan 1995). The project involves crafting interactive board game elements that leverage AR technology to enhance gameplay, creating a visually captivating and dynamic experience for players. As previously touched on, ‘The Blend’ as it is referred to, is the concept of finding a balancing point between integration and redesigning the older game elements and pieces. The goal is to create elements of physical tools that feel familiar to older audiences who recognize and enjoy the original game, while also providing a new spin on those elements that will be able to reach and draw in a demographic of new players who are familiar with a more modern form of entertainment.

Freezing of gait is a common, treatment-resistant symptom of Parkinson Disease that cancause falls and reduce quality of life. Previous research has demonstrated benefits ofcueing on freezing of gait. However, no effective cue modality has been easily adaptableto daily, real-world use. Visuospatial augmented reality cues are promising in this regard,but have not shown reduction in freezing of gait, possibly due to cue design andlimitations of past augmented reality headsets. The purpose of this study was to developand test a versatile new augmented reality cue on a recently released, field-leadingaugmented reality headset. Also developed were several novel visuospatial cue activationstrategies which were compared to established modalities.The cue was tested in a population with Parkinson Disease and freezing of gait.While no single strategy significantly reduced freezing of gait across the cohort, eachparticipant had on average at least two cueing strategies which significantly reduced timespent freezing compared to a physical cue and control (no cue). Those strategies alsoimproved mean stride length, a marker of gait health. Survey results indicate that themajority of the cohort found the cue to be effective and, with an improved headset, woulduse the cue in their daily life.

Shopping is important in our life, especially food shopping. Almost every family needs to go shopping once per week and spend lots of time and energy on it. This project proposes the design of an application to help people have a better shopping experience by saving time, saving energy, and saving money. The application combines online shopping and in-person shopping in a new shopping mode. People don’t need to pay the expensive delivery fee and wait a long time to get food which could be unfresh. And they also don’t need to spend lots of time at the supermarket, which could cost lots of time by walking around and finding items. This application can solve all shopping problems people might face today. My design intends to help people have a better shopping experience and save precise time on weekends.

General Physics laboratory experiments can be time-consuming to set up for each class.Additionally, students often have a specific room and limited time to conduct the experiment and draw conclusions. One advantage of moving the labs to an augmented reality setting is that setup time can be lowered. The main motivation, however, would be to allow students more time to explore the experiments on their own outside of class. Because the laboratory experiment is a computer simulation, it could provide students with options not available in a physical lab, for example, one could alter the acceleration due to gravity. With added capabilities, augmented reality laboratory experiments could be used either as a replacement for, or an extension of, traditional laboratory experiments. This paper explores the feasibility of using augmented reality to replace Physics laboratory experiments. A one-dimensional motion Physics laboratory experiment was recreated in the Unity game engine and deployed to the Microsoft Hololens 2. Study participants worked through instructions which had them collect data and perform calculations based on that data. The experiment was conducted as an exploratory study to get feedback on the user interface and how well participants were able to complete the tasks, and to gauge the general response to using augmented reality for a Physics laboratory experiment. Based on early findings, an augmented Physics laboratory experiment appears to provide a suitable replacement, but the provided material and available toolset contribute to the effectiveness of the experience.

Analysts often work with large complex non-quantitative datasets in order to better understand concepts, themes, and other forms of insight contained within them. As defined by Pirolli and Card, this act of sensemaking is cognitively difficult, and is performed iteratively and repetitively through various stages of understanding. Immersive analytics has purported to assist with this process through putting users in virtual environments that allows them to sift through and explore data in three-dimensional interactive settings. Most previous research, however, has focused on quantitative data, where users are interacting with mostly numerical representations of data. We designed Immersive Space to Think, an immersive analytics approach to assist users perform the act of sensemaking with non-quantitative datasets, affording analysts the ability to manipulate data artifacts, annotate them, search through them, and present their findings. We performed several studies to understand and refine our approach and how it affects users sensemaking strategies. An exploratory virtual reality study found that users place documents in 2.5-dimensional structures, where we saw semicircular, environmental, and planar layouts. The environmental layout, in particular, used features of the environment as scaffolding for users' sensemaking process. In a study comparing levels of mixed reality as defined by Milgram-Kishino's Reality-Virtuality Continuum, we found that an augmented virtuality solution best fits users' preferences while still supporting external tools. Lastly, we explored how users deal with varying amounts of space and three-dimensional user interaction techniques in a comparative study comparing small virtual monitors, large virtual monitors, and a seated-version implementation of Immersive Space to Think. Our participants found IST best supported the task of sensemaking, with evidence that users leveraged spatial memory and utilized depth to denote additional meaning in the immersive condition. Overall, Immersive Space to Think affords an effective sensemaking three-dimensional space using 3D user interaction techniques that can leverage embodied cognition and spatial memory which aids the users understanding.

Micro-light-emitting diode (microLED, micro-LED, μLED) is an emerging display technology based upon high-density arrays of micron-sized III-nitride (or III-V) LEDs. Small form factor displays, including wearables (smartwatches) and AR (augmented reality) glasses will benefit most from microLED. The paradigm shift from LEDs for solid-state lighting to microLEDs for displays introduces new research challenges. MicroLEDs become difficult to process as dimensions reduce. Mesa diameters as small as 1 μm are desirable for maximum resolution and small sizes in AR displays. However, layer-to-layer alignment errors during lithography prevented researchers from fabricating microLEDs below 5 μm when using standard LED processing approaches. This thesis presents a self-aligned processing technique developed to circumvent these issues. The result is the smallest microLED ever published at 1 μm diameter. This milestone enables new microLED device research at the size ranges most targeted for AR displays. Due to the different usage and requirements for microLEDs in displays versus LEDs in lighting, new measurement systems for characterizing device performance were constructed. Key considerations when characterizing on-chip microLED electroluminescence are discussed. As dimensions are reduced in microLEDs, new size-dependent physical effects appear. It is well known that microLED efficiency decreases as the diameter shrinks due to increased non-radiative surface recombination. However, results comparing red, green, and blue microLEDs show increasing the indium fraction in InGaN limits surface recombination through enhanced carrier localization. An added effect in small microLEDs is improved extraction efficiency from sidewall reflections redirecting photons within the extraction window. To separate competing size-dependent contributions from sidewall recombination and sidewall reflections, optical models are necessary. A combined approach uses FDTD to model interactions with near-wavelength-sized patterned sapphire features beneath the mesa.

Automated vehicles pose challenges in various research domains, including robotics, machine learning, computer vision, public safety, system certification, and beyond. These vehicles autonomously handle navigation and locomotion, often requiring minimal user interaction, and can operate on land, in water, or in the air. In the context of aircraft, one specific application is Automated Aerial Refueling (AAR). Traditional aerial refueling involves a "tanker" aircraft using a mechanism, such as a rigid boom arm or a flexible hose, to transfer fuel to another aircraft designated as the "receiver". For AAR, the boom arm may be maneuvered automatically, or in certain instances the tanker may remotely pilot the receiver to ensure station keeping during the refueling process. Due to latency issues, ground control is impractical. Any AAR system must perform rapid and precise relative pose calculations, operating in near real-time (approximately 10 Hz) with an accuracy of around 10 centimeters when the receiver is 30 meters from the tanker.Utilizing stereo vision for relative pose estimation necessitates high-resolution images to achieve this level of accuracy. Given the large resolution, data processing must be extremely fast. The Iterative Closest Point (ICP) method estimates a pose from images by establishing point correspondences and calculating the relative pose between two point clouds. To accelerate ICP, two enhancements contribute to the runtime reduction: using Delaunay triangulation for finding correspondences and parallelizing each step of the ICP process. Since ICP is iterative, caching correspondences at each iteration enables the nearest neighbor search to start from a point likely to be close to the true closest point.Experiments with accelerated ICP demonstrate a speedup on the order of 103 compared to an O(n2) nearest neighbor method and 102 compared to a k-d tree. When tested against other parallel ICP implementations, the accelerated Delaunay ICP matches or outperforms them in terms of timing results. Notably, Delaunay ICP does not require parameter tuning, leading to more accuracy pose estimations compared to other parallel ICP implementations. To further validate the correctness of Delaunay ICP, real and virtual simulations of an AAR scenario provide a robust test-bed for stereo vision and point registration techniques.

U.S. federal laws mandate that mining companies ensure a safe workplace, implement approved training programs, and promptly report work-related injuries. The mining industry's commitment to innovation reflects a history of adopting advancements to enhance environmental sustainability, workplace safety, and overall productivity, while simultaneously reducing operational costs. This thesis proposes the integration of Augmented Reality (AR) technology and digital applications to enhance the surface mining industry, presenting two innovative solutions: an AR Training System and an Operational Digital System. These business solutions have been developed and applied at a surface mine in the southwest of the US, having the potential to improve the mining industry by enhancing safety, training, operational efficiency, and data-driven decision-making, which comprehends a significant step toward a more sustainable, effective, and technologically driven mining sector, contributing to the industry's evolution and growth. The AR Training System leverages Microsoft´s Power Platform and HoloLens 2 capacities to provide operators with immersive and step-by-step training guides in real working conditions for Dozers, Motor Graders, and End Dump trucks. These AR guides combine 3D models, videos, photos, and interactive elements overlapping mining equipment to enhance learning and safety. The system also offers an efficient approach to data collection during operator training, which has the potential to modify the training guides based on user performance. On the other hand, the Operational Digital System addresses the industry's operational challenges. It streamlines the pre-operation inspection process, tracks equipment status, and accelerates defect identification, shift timing, delays, and loaded tonnage. The system offers a holistic approach to mining operation optimization, facilitating data sharing and management among different departments, enhancing collaboration, and expediting maintenance processes.

Immersive technologies are widely used in interactive media art and storytelling today. A high-quality virtual environment and avatar can provide the audience with a strong sense of presence and ownership of the virtual body, evoking empathetic outcomes from the experience. However, the majority of immersive art we encounter today shows a strong favoring of visual and auditory sensation over interactions. They provide the audience with a limited number of ways to participate in an immersive narrative, leaving them a sense of being an “outsider” that breaks the connection between the audience’s virtual avatar and the virtual environment. Starting with Nicolas Bourriaud’s relational aesthetics for art creation and taking human relations and their social context as the main conceptual and practical point of departure for experience design, I develop artistic and technical strategies for creating immersive experiences that place the audience at the center to let them activate the narrative content and cultural meanings. My approach is to provide the user novel ways to relate themselves to what they see on a screen, and to allow for direct participation and engagement. Through an increased level of interactivity, co-presence and plausibility in immersive environments, my work aims to help the audience develop new relationships with other people, with virtual entities, and with AI agents. The hybrid systems introduced in this thesis describe novel ways of creating relational experiences by techniques learned from different domains such as soft robotics, quantum computing and social science. I also demonstrate authoring interface design that can be more widely adopted by the creative community to overcome the challenges often faced in immersive and interactive narrative compositions.

In recent years, Augmented Reality (AR) glasses have emerged as groundbreaking wearable devices, enabling a harmonious integration of virtual information with the real world. However, challenges remain in enhancing the optical systems that underpin AR glasses, including resolution and field of view limitations. This research focuses on leveraging metasurfaces, composed of subwavelength nanostructures, to design and optimize advanced optical components for AR glasses. By tailoring the properties of metasurfaces, such as amplitude, phase, and polarization control, we aim to overcome limitations and enhance the resolution, field of view, and overall visual performance of AR glasses. To achieve truly immersive AR experiences, improving the optical performance is essential. This involves minimizing aberrations, enhancing contrast, reducing glare, and optimizing light transmission efficiency. Metasurfaces offer unique capabilities for precise manipulation of light at the nanoscale, enabling the creation of highly efficient and compact optical components. By integrating metasurface-based elements, such as gratings, lenses, beam splitters, and polarization controllers, we can enhance resolution, expand the field of view, and improve overall optical performance. Furthermore, metasurfaces offer multifunctionality and the potential for dynamic control. By combining multiple functionalities within a single metasurface-based element, the complexity of optical systems can be reduced, leading to more compact and lightweight AR glasses. Additionally, incorporating tunable or switchable elements within metasurfaces enables real-time control of optical properties, allowing for adaptation to changing environmental conditions and advanced functionalities like adaptive focus and depth perception. This research also explores the challenges of small eyebox in Maxwellian view systems, which restricts user head movement and comfort. Innovative methods utilizing input angle modulation at the metasurface optical element (MOE) are proposed to increase the eyebox size while maintaining optical performance. The study includes the design and optimization of metasurface gratings, algorithm development for optimizing optical elements, implementation of a high-resolution full-color prototype, and investigation of imaging techniques like ultrafast light field tomography (LIFT). By combining these research efforts, this study aims to significantly advance the design and optimization of metasurface-based diffractive waveguide systems for AR applications. The outcomes have the potential to revolutionize the field of augmented reality, pushing the boundaries of optical resolution, addressing limitations, and enhancing the overall user experience. The findings contribute to the advancement of AR glasses, enabling new possibilities in industries such as education, entertainment, healthcare, and beyond.

With the increasing adoption of Augmented Reality/Virtual Reality (AR/VR - XR) systems, security, privacy, and quality concerns attract attention from both academia and industry.We identify four key problems in XR applications and make contributions respectively.For XR Quality of experience: AR applications suffer from spatial inconsistency where virtual objects "drift or "jump" in space. We develop an automated method to evaluate the spatial inconsistency of AR applications. Web-based XR suffers from long latency in downloading virtual content to be displayed on a web browser. We develop a technique to divide up and prioritize virtual objects to be downloaded in a manor to minimize the time to first correct image. For XR Security: Head mounted displays track the user's head pose at all times in order to render content correctly. We show that these head poses can be used to help infer user hand-typed words in order to steal private information. Multi-use AR applications construct a "shared-state" using image sequences and GPS data. We show that these shared-states can be poisoned allowing for the reading and writing of information to and from arbitrary locations by a malicious user.

Augmented Reality (AR) displays show virtual information spatially in the physical world. For example, in the surgical domain, AR displays can overlay medical imaging on top of the patient to help surgeons distinguish healthy tissue from cancerous tissue. This affordance of adding visual information near the area of action has led to a gamut of AR applications with one caveat: their design assumes a flawless alignment between the virtual information and its physical context. If this alignment fails, the system might mislead users by presenting the virtual information out of context.Fortunately, technical advancements in tracking, calibration, and optics have helped improve and create a more seamless augmented reality experience; however, even in these improved systems, alignment failure will lead to user-task performance errors. In my dissertation, I argue that AR interface design needs to account for such misalignments and limitations of AR technology. First, I investigated the case of a situated checklist for cardiopulmonary resuscitation (HoloCPR), which helped deliver accurate guidance even in the presence of tracking errors. Then, I investigated a telementoring system for trauma procedures (ARTEMIS) which helped remote surgeons communicate with novices despite a constant registration error. Finally, I showed that slight optical deviation can drastically impact overall user-task performance and cognitive workload, even in an idealized environment with perfect tracking and calibration.I sum the findings of this final piece, along with the systems designed and developed in this dissertation, to point in the direction of a new type of AR interface building block: contextual aids and how they may have significant implications for AR interface design for surgical procedures, paving the way for the development and integration of more advanced AR systems in surgical practice.

This thesis proposes a novel interactive immersive analytics tool and methods to interrogate the cancer patient cohort in an immersive virtual environment, namely Virtual Reality to Observe Oncology data Models (VROOM). The overall objective is to develop an immersive analytics platform, which includes a data analytics pipeline from raw gene expression data to immersive visualisation on virtual and augmented reality platforms utilising a game engine. Unity3D has been used to implement the visualisation. Work in this thesis could provide oncologists and clinicians with an interactive visualisation and visual analytics platform that helps them to drive their analysis in treatment efficacy and achieve the goal of evidence-based personalised medicine. The thesis integrates the latest discovery and development in cancer patients’ prognoses, immersive technologies, machine learning, decision support system and interactive visualisation to form an immersive analytics platform of complex genomic data. For this thesis, the experimental paradigm that will be followed is in understanding transcriptomics in cancer samples. This thesis specifically investigates gene expression data to determine the biological similarity revealed by the patient's tumour samples' transcriptomic profiles revealing the active genes in different patients. In summary, the thesis contributes to i) a novel immersive analytics platform for patient cohort data interrogation in similarity space where the similarity space is based on the patient's biological and genomic similarity; ii) an effective immersive environment optimisation design based on the usability study of exocentric and egocentric visualisation, audio and sound design optimisation; iii) an integration of trusted and familiar 2D biomedical visual analytics methods into the immersive environment; iv) novel use of the game theory as the decision-making system engine to help the analytics process, and application of the optimal transport theory in missing data imputation to ensure the preservation of data distribution; and v) case studies to showcase the real-world application of the visualisation and its effectiveness.

This study explores whether augmented reality (AR) is an effective form of visual communication through poster design. Effectiveness is measured here by how engaging viewers find posters to be and how well they are able to recall information presented on the posters.Secondary research was conducted to understand what affordances AR provides that may improve engagement and information recall. User interviews and card sorts established a framework for how viewers think about augmented reality techniques—namely whether or not the augmentation breaks the original frame of reference and how relevant the augmentation is to the poster’s message.Using this framework, four tests were conducted using experimental groups, which were exposed to augmented reality communications, and control groups, which received traditional print communications. These tests included user observations, interviews, and surveys to measure the viewers’ perceived level of engagement with the communications as well as their ability to recall information immediately following the viewings and one week later.The study finds that the use of augmented reality in poster communication does significantly increase a viewer’s perceived level of engagement with the communication, but also significantly decreases their ability to recall information from the poster. The use of augmented reality distracts the viewer and interrupts their ability to encode information at a semantic level, which is a process that improves information recall and is needed to create meaning. The study also finds that the bio-cost of viewing AR posters is currently large enough that viewers are unlikely to engage with them by choice.

Nowadays, people prefer browsing images and videos on their phones rather than reading a book (Thain 2018). Even if they decide to read something, they are still heavily distracted by electronic devices. In my thesis, I’m trying to balance the visual and text information and help people to stay more focused while reading. My solution: Focus is an AR glasses system that recognizes the contents users are viewing and records users’ reading and image browsing time. And users have to earn image/video time by reading the text. The glasses block images/videos if they run out of image time.

The demand for technology integration has presented teachers with new pedagogical choices for designing engaging learning experiences for students. These technology choices are becoming abundant and varied, confronting teachers and administrators with determining how effective these methods are for teaching. This research was focused on one of these choices, whether including mixed reality (MR) technologies enhanced the learning experience for college students in a field-based course. The experiential learning theory (Kolb, 1984) provided the framework for understanding the learning process through the experiential learning cycle; a cyclical process of starting with a concrete experience, reflecting on that experience, formulating new concepts, then applying what was learned. This theoretical framework was used to explore the research questions of how MR technology might enhance the learning process, enrich the reflections process, and determine if the students perceived these MR enhancements as useful. This constructivist process for learning was the basis for developing the MR field guide which provided the tool for examining the student experience while they participated in a soil science field study course. This research followed a constructivist-interpretive paradigm using a qualitative, exploratory case study methodology Merriam & Tisdell (2015). Interviews and surveys were used to illuminate the experience the students were having while using the MR field guide. Key findings indicated that the inclusion of the MR field guide has the potential to improve the learning experience and the reflective observation needed to influence learning. The research found that the MR technology facilitated by the MR field guide had both advantages and challenges in enhancing the learning experience. Student participants felt MR technology provided enhanced learning opportunities if certain challenges were overcome. The insights gained from the findings of this study will add to the knowledge for developing more effective teaching strategies when using MR technology and provide guidance to future efforts for technology integration in environmental education.

In the process of linguistic landscaping, spaces are transformed into textual and semiotic representations linked to particular social uses, cultural meanings, and historic narratives. Recently, digital technology has been used to create additional layers of semiotic representation in linguistic landscapes. This dissertation investigates multiple layers of digital and physical representation at six United States’ park spaces, with a particular focus on heritage tourism sites, and analyzes the social meaning and narratives of tourism constructed both by individual layers and within their relationships. Photo-based methods are used to collect data, both by documenting the representations of sites as filtered through the AR platform, Niantic’s Pokémon GO, as well as through photographing the physical sites in-person. This process of ground truthing revealed the erasure of Indigenous histories, as well as amplification of eurocentric histories, which was perpetuated by both the physical and digital layer. The results also revealed a simplification of representations via the digital layer, as well as the role of corporate mediation in regulating space representation. These results are significant in revealing the role of augmented reality in portraying a particular sanitized and gamified version of the spaces. This project also suggests a need within linguistic landscaping research for more engagement with the interdisciplinarity of landscape studies as a whole, particularly, in a US parks context, for gaining a more precise and holistic view of the complex histories and contexts that are obscured by both physical and digital representations.

The internet and mobile devices allow us to consume digital data at every corner. We also encounter data in the real world between these interactions with the virtual world. When we see movie posters, pass street signs, or fill out a medical form, we observe data in the real world instead of the virtual world. The rise of augmented reality (AR) technologies allows us to consider how we can improve interactions with this real-world data. For example, consider a restaurant-goer who wishes to find which items are popular on the menu they are looking over. Currently, the person must look up the ratings online themselves. In this thesis, we propose methods that enable the person to view the menu with their mobile device and query for the menu item ratings by interacting directly with the real-world view of the menu.The work in this thesis enables systems that capture real-world data through an AR device, enhance the real-world data with data from remote sources, and allow a user to query real-world data in augmented reality interactively. The first part of this thesis describes a visual querying grammar and the implementation of a system for querying tabular data in the real world. The second part of this thesis showcases a declarative framework for developing applications that join real-world data with data from remote stores.

Emerging mobile systems such as robots or Augmented Reality glasses increasingly interact with their physical surroundings. This means they require efficient computing platforms to continuously process a high volume of sensory information in a limited-energy battery-powered device. Highly specialized Domain-Specific System on Chips (DSSoCs) have been recently proposed as a solution to provide this efficiency. However, although efficient, the complexity of DSSoCs driven by a high count of hardware intellectual property (IP) blocks results in a long development time, concretely a long design exploration and implementation time. This is because a high IP count enlarges the design space, makes the search for optimal design difficult, and thus lengthens the exploration time. In addition, each new IP in the system increases the implementation time, i.e., the time associated with converting an algorithm to a hardware IP, verification, and integration. These two problems only worsen as the future system’s intelligence and, thus, computational need expands. This dissertation demonstrates how to mitigate the DSSoC’s long exploration and implementation time by employing efficient resource selection and resource management techniques. We demonstrate how to lower the exploration time using an agile design space exploration (DSE) framework that efficiently selects system resources. Similar to other DSEs, our framework consists of a simulator and an exploration heuristic. In contrast to the state-of-the-art simulators that are either too slow or inaccurate and thus infeasible for sufficient coverage of DSSoCs’ large design space, our simulator is highly suited for DSSoCs as it combines the agility of analytical models and the accuracy of the phase-driven models. Our result shows that this methodology speeds up the state-of-the-art transactional models by 8400x while only incurring a small 1.5% error for a host of complex SoCs. Our DSE also lowers the exploration time by deploying an agile search heuristic that efficiently navigates DSSoCs’ large design space. In this dissertation, we highlight two features for an efficient search heuristic, namely 1) joint-optimization capability to exploit cross-boundary optimizations and 2) architectural awareness to prevent blind traversal of the design space. State-of-the-art solutions either lack the former or the latter. In contrast, our methodology combines the two and speeds up the convergences of the baseline, i.e., the classic simulated annealing (SA) and more modern Multi-Objective optimistic Search (MOOS) by 62× and 35×, respectively. It also improves their Quality Gain and Pareto Hyper Volume. We also demonstrate how to lower the implementation time by replacing high-effort IP customization solutions with low-effort environment-aware resource management ones. Our solution uses the physical environment’s heterogeneity, a prominent characteristic of the mobile edge domain, to its advantage and dynamically manages system resources to improve efficiency. To this end, first, we show that various spatial heterogeneity factors, such as environment congestion, impact the processing payload. Then, we establish that designs that ignore this heterogeneity incur system-level performance and energy degradation and thus require specialized IPs to solve their efficiency problem; However, introducing IPs increases the implementation time and effort. Thus, to address this problem, we provide a resource management runtime that dynamically exploits said compute-environment synergy and improves the system efficiency without introducing customized IPs. We implement our runtime in Robot Operating System (ROS) middleware and evaluate it for autonomous drones without the loss of generality. We compare our runtime with a spatially oblivious system, typical of traditional commercial deployments, whose parameters are statically set at design time. We show that exploiting spatial heterogeneity leads to 4.5x improvement in mission time, 4x improvement in energy efficiency, and 36% reduction in CPU utilization. The contributions made in this dissertation will likely have a long-term impact as, with the end of Moore’s law, highly specialized DSSoCs have shown promise to address general systems’ lack of efficiency. Thus, understanding and further providing techniques to mitigate DSSoCs’ design and development issues are of high importance.

Orientation Device is a tool for understanding the other towards recognizing alternative possibilities, for care and compassion, for expanding our culturally and politically bounded mindset, a tool of vital nausea and questioning compulsory heterosexuality. The work is a series of augmented reality (AR) experiences for mobile device that allow the audience to participate in documentary queer performances in any private or public setting. These immersive experiences challenge our perception of space. The LGBTQIA+ community is often disoriented within heteronormative spaces, and this work reverses that dichotomy by disorienting the audience. As a cisgender creator, I invite queer performers, artists, poets, and thinkers who express their identity in their creative practices. As the AR medium is widely distributable, I want to give the participants the potential opportunity to present themselves to a wide international audience through the poetics of augmented reality and documentary video holograms.

The purpose of this causal-comparative study was used to analyze the hypothesized differences among inclusion teachers’ sense of self efficacy (TSES) based on their highest level of degree completion (bachelor’s, master's, master's plus). In an era of educational reform, students with specific learning disabilities (SLD) continue to lag behind regular education counterparts in all schooling environments. Schools in the southwestern region of Tennessee are now servicing a growing number of SLD-diagnosed students with a paucity of research addressing teacher's impact on SLD literacy scores. Participants in the study comprised 59 English language arts (ELA) inclusion teachers in 12 public middle school settings. To address this gap in the research, data were examined using a one-way analysis of variance (ANOVA). Survey Monkey and TSES long form was used to collect demographic and professional information. Results of the one-way ANOVA revealed no statistically significant differences among inclusions teachers with different levels of education. Future recommendations for research in virtual reality (VR) and augmented reality (AR) can enhance instructional practices while building teacher's confidence to motivate students' interest in ELA achievement in all public-school learning environments.

Ruminations is an experimental exhibition which seeks to explore the integration of technology into traditional art. By combining traditional woodblock prints with augmented reality, Ruminations attempts to evoke empathy from the viewer regarding the topic of mental illness. Each AR experience illustrates, though imagery and animation, a different mental illness or psychological phenomenon. The viewer sees each experience separately over a printed background, expanding on the traditional frame and reaching into the world around us.

This research is motived by the study of dynamics, especially in experimentation and Structural Health Monitoring (SHM) methods. It is important that inspectors maintain awareness of test structures while observing sensor data and inputting control. Humans receive a large amount of information from vision, and this feedback is crucial to inform decision making. Human-computer interaction (HCI) provides valuable data and information but separates the human from reality as it is necessary to look away from the region of interest to view information on a separate device. Additionally, sensor data does not collect experiment safety, quality, and other contextual information of critical value to the operator. Safety, informed decisions, and control by engineers in the laboratory or field would be increased if they could maintain focus on their environment while evaluating the data their senses are not equipped to obtain (i.e., sensors and processing equipment). Furthermore, humans could advance robotic and machine manipulation if informed on real-time of the consequences of their decisions during human manipulation. To solve this problem, this research provides humans with Augmented Reality (AR) tools for engineering tasks. AR provides additional information to the AR user via head-mounted device (HMD), which allows the user to operate and observe the physical space without impedance. The two primary areas of focus in AR development in this MS thesis are visualization and control, where several applications are developed for specific use in engineering tasks. Elements of visualization and control are present in each of the applications. The primary application provides an interface for sensor feedback in AR. This inspires an AR interface for control of actuators in vibratory experimentation. The application is developed to plot sensor data in an interface complete with voltage, frequency, and duration controls for vibration generation. Implementing robots into cyber-physical systems for SHM promotes human capabilities that are improved by robot capabilities. Intuition often allows human workers to solve different tasks faster than robots, and when these human capabilities are coupled with the repeatability and endurance of robots then full potential can be realized. Two applications are developed for feedback and control of robotic arms in AR for the purpose of sensor deployment. The two arms are the Cyton Alpha 7-degree-of-freedom (DOF) arm and the Kinova Gen3 7DOF arm. This MS thesis also presents an AR application for an acoustic SHM method, deemed tap testing, which is used to detect signs of deterioration in structural surfaces through nondestructive means. The system is setup on a mobile robot titled Brutus, which is equipped with a sonar sensor to measure the distance between the robot and test surface. Experiments are conducted for verification of the developed applications, and the results of the reported experiments indicate that augmenting the information collected from sensors in real-time along with interfaces for control narrows the operator’s focus for more efficient and informed task conduction. This thesis considers the importance of human-centered framework where often experts in SHM prefer to be present to make decisions based on their own cognition, which can be coupled artificial intelligence and automation without solely depending on it. The research solves a problem with HCI where the operator experiences gaze distraction when attempting to monitor data and dynamic events. AR provides additional information to the AR user via HMD, which allows the user to continue to operate and observe the physical space without impedance. The results enable the research community to design, program, and examine new AR applications interfacing sensor feedback and control with real structures and environments.

Spatial referencing denotes the act of referring to a location or an object in space. Since it is often essential in different collaborative activities, good support for spatial referencing could lead to exceptional collaborative experience and performance. Augmented Reality (AR) aims to enhance daily activities and tasks in the real world, including various collaborations and social interactions. Good support for accurate and rapid spatial referencing in collaborative AR often requires detailed environment 3D information, which can be critical for the system to acquire as constrained by current technology. This dissertation seeks to address the issues related to spatial referencing in collaborative AR through 3D user interface design and different experiments. Specifically, we start with investigating the impact of poor spatial referencing on close-range, co-located AR collaborations. Next, we propose and evaluate different pointing ray techniques for object reference at a distance without knowledge from the physical environment. We further introduce marking techniques aiming to accurately acquire the position of an arbitrary point in 3D space that can be used for spatial referencing. Last, we provide a systematic assessment of an AR collaborative application that supports efficient spatial referencing in remote learning to demonstrate its benefit. Overall, the dissertation provides empirical evidence of spatial referencing challenges and benefits to collaborative AR and solutions to support adequate spatial referencing when model information from the environment is missing.

The purpose of the study was to examine the impact of using AR in the Computer Architecture unit for male 11th grade students in a school in the eastern area of Arar City in Saudi Arabia through monitoring its impact on student achievement and students' perceptions of the learning environment. Two research questions are explored: What is the effect of using AR on student achievement, and what are students' perceptions of the learning environment when they use AR? Two instruments were used to collect the data in this study: an achievement test taken from the official teacher book issued by the Ministry of Education in Saudi Arabia and the Technology-Rich Outcomes-Focused Learning Environment Inventory (TROFLEI) modified questionnaire "actual form." Statistical analyses employed to answer the first research question included an independent-samples t-test and descriptive statistics. To investigate the second research question, descriptive statistics and a paired t-test were used. These results from the first question indicate a statistically significant difference (p < 0.05) between the two groups' mean values: the students who used AR achieved a higher level of learning compared to the students who learned in the traditional way. The study found that using AR helped the students to increase their achievements through many aspects, one of which was being able to feel in contact with objects and events that were physically out of their reach. In addition, AR offered a safe environment for learning and training away from potential and real dangers. The results for the second research question show statically significant increases in seven out of eight TROFLEI scales. This suggests that there was a positive feeling among the students regarding the teacher's interaction and his interest in providing equal opportunities to the students to answer the questions.

No abstract required.

This research project outlines the mental health benefits of art-making and establishes exposure to artistic domains as essential to the art-making process. Understanding the role of intent in behavioral change established the need for external motivators, such as gamification, in the adoption of new habits. Pokémon Go players in Cincinnati, Ohio were studied to understand the relationship between virtual gameplay and physical spaces. Feminist geography and the principles of meaningful gamification informed the design solution, a location-based augmented reality game (LARG) that utilizes artwork by women as elements of gameplay to mitigate patterns of underrepresentation in the arts. The research showed that a LARG would not only increase engagement for those new to the arts but also for those already engaged by providing a new perspective on familiar experiences; travel between locations of gameplay also offers an additional landscape for cultivating meaningful engagement.

Augmented reality (AR), as a key component of the future metaverse, has leaped from the research labs to the consumer and enterprise markets. AR optical see-through (OST) devices utilize transparent optical combiners to provide visibility of the real environment as well as superimpose virtual content on top of it. OST displays distinct from existing media because of their optical additivity, meaning the light reaching the eyes is composed of both virtual content and real background. The composition results in the intended virtual colors being distorted and perceived transparent. When the luminance of the virtual content decreases, the perceived lightness and brightness decrease, and the perceived transparency increases. Lightness, brightness, and transparency are modulated by one physical dimension (luminance), and all interact with the background and each other. In this research, we aim to identify and quantify the three perceptual dimensions, as well as build mathematical models to predict them. In the first part of the study, we focused on the perceived brightness and lightness with two experiments: a brightness partition scaling experiment to build brightness scales, and a diffuse white adjustment experiment to determine the absolute luminance level required for diffuse white appearances on 2D and 3D AR stimuli. The second part of the research targeted at the perceived transparency in the AR environment with three experiments. The transparency was modulated by the background Michelson contrast reduction in either average luminance or peak-to-peak luminance difference to investigate, and later illustrated, the fundamental mechanism evoking transparency perception. The first experiment measured the transparency detection thresholds and confirmed that contrast sensitivity functions with contrast adaptation could model the thresholds. Subsequently, the transparency perception was investigated through direct anchored scaling experiment by building perceived transparency scales from the virtual content contrast ratio to the background. A contrast-ratio-based model was proposed predicting the perceived transparency scales. Finally, the transparency equivalency experiment between the two types of contrast modulation confirmed the mechanism difference and validated the proposed model.

Phenomenal Things is a comical look into the daily lives of Internet of Things (IoT) artifacts and their experiences as social beings in cyberspace. This Augmented Reality (AR) experience presents a storyworld set in the digital realm where the digital personas of IoT artifacts are engaged in activities normally invisible to humans such as information extraction, learning, talking to each other and communicating with other "things" online. By wearing a head- worn display (HWD), users will encounter anthropomorphized IoT artifacts going about their daily lives and come to understand these characters as digital beings with social lives. Placed inside of cyberspace, participants will find themselves within a circle of anthropomorphized IoT devices in dialogue with one another, as they welcome a new light bulb to their network. As participants move about the AR actors, proximity to each character will cause the participant to "friend" that character. "Friending" in this case means to get close to and influence the version of the story being told by changing the social network of the character. With this work I intend to create a mesmerizing yet subtly-interactive experience using proxemics to create an interactive narrative where participants can create emotional bonds with the AR actors in this immersive theater experiment.

This is a scholarly personal narrative about the author’s three-year journey leading to the development of the Recycling Distraction, a website designed to raise awareness of the shortcomings of municipal recycling. Based on research by environmental historians, sociologists, and economists, the Recycling Distraction presents a critical perspective on America’s reliance on municipal recycling for managing post-consumer waste. Through the website, the author challenges the longstanding corporate practice of individualizing environmental responsibility to avoid anti-litter regulations, and argues that companies should be held responsible for their proportionate share of environmental damage caused by single-use disposable products. This paper documents the author’s experience learning the fundamentals of user-experience (UX) research as an MFA candidate in the Department of Design. During the journey, the author navigates the landscape of sustainability programs at The Ohio State University, and searchers for opportunities to conduct UX research to support the development of circular-economy initiatives at the institution. Following a series of exploratory design activities, including: (1) conducting user-experience inquiries into recycling-related problems; (2) designing digital solutions aimed at improving recycling participation; (3) reviewing recycling and circular-economy literature; and (4) reflecting upon on his experiences, the author redirects his efforts toward raising awareness of recycling’s failure to reduce waste and prevent pollution, which culminates with the development of the Recycling Distraction project.

Augmented Reality (AR) is an interactive technology that delivers rich and immersive computer-generated perceptual information overlaid onto the real world. AR technology continues to improve over time, potentially enabling an "always-on" AR future, where wearable AR devices are as comfortable to wear as glasses and have an all-day battery life. This concept has been termed "Pervasive Augmented Reality", in which AR is the predominant form of personal computing, instantly accessible, constantly providing information and available to assist in everyday tasks.Pervasive AR represents a shift in how, when, and where we use computers, but there are few guidelines for how to effectively create these types of systems. Current research focuses on single-purpose use cases, designed for specific tasks and environments, such as navigation or maintenance. They rarely consider AR as a mobile experience that can be taken anywhere, where computing happens continuously and persistently. This dissertation addresses that insufficiency by analyzing the design and evaluation of AR artifacts for 3 use cases: AR Recommender Systems, AR Language Learning, and AR Multitasking. We conduct our investigation in 3 parts. 1) What benefits do AR applications provide over non-AR based systems? 2) What additional inputs and signals can AR applications benefit from? and 3) How do we evolve current AR systems towards a pervasive AR future? Our work makes contributions through empirical user studies, prototype systems, and the development of new interaction techniques, revealing insights into the tools and techniques needed for developing pervasive AR, as well as the opportunities and new possibilities it enables over existing technologies.

Coordination among teams or pairs is vital to completing cooperative problem-solving tasks. Moreover, trust among team members enhances performance in cooperative settings, but how coordination and trust relate to each other among partners who have no past history together is uncertain. Therefore, this study tested the relationship between coordination performance (accuracy and completion time) and levels of trust (cognitive and affective) in an augmented reality tangram matching task. Pairs of young adult participants (N = 104) entered a laboratory to play a game wherein “directors” described target shapes to “matchers” who attempted to match targets in a smartphone-displayed grid. Participants answered trust surveys after each of three game rounds. The effects of task-role on self and partners’ trust perceptions were explored using an actor-partner interaction model. Cognitive trust was positively associated with accuracy but not completion time, implying the relevance of cognitive trust on coordination task performance among zero-history dyads. Furthermore, directors’ cognitive trust towards partners influenced matchers’ change in cognitive trust toward directors, but not vice versa. This suggests that differences between task-roles may lead to reciprocated trust from one, but not both, partners in a dyad. The study discussed communication behaviors related to different levels of trust, the grounding process, and incongruencies between task-roles, as to propose future research avenues.