Search

ASD is a neurodevelopmental condition characterised by impairments in social communication and interaction and restricted or repetitive patterns of behaviour. Early intervention is recommended to produce the best outcomes for children with ASD. Clinicians therefore use specific therapies that most effectively improve outcomes for children. An intervention that is widely used by teachers and therapists is role play. Role play is a form of pretend play which is widely used by educators and therapists, to improve a variety of skills in children with ASD, including social and communication skills and Theory of Mind (ToM). ToM is the ability to attribute mental state to others, or in other words, the ability to 'put yourself in someone else's shoes'. In recent years, digital interventions to support learning for children with ASD which use interactive game-based technology have been increasing. Augmented reality (AR) is one such technology that is beginning to show promising results. AR is an advanced technology, commonly using visual cues, that expands our world by superimposing digital content on top of reality. Several unique features of AR have been shown to support learning for children with Autism Spectrum Disorders (ASD). This study explores the design and creation of an innovative augmented reality pretend play intervention for children with autism spectrum disorders (ASD), through a user centred design process (UCD), also known as human centred design. A conceptual system framework was presented for researchers to use to create digital pretend play interventions for children with ASD. The conceptual system framework consisted of user requirements, instructional guidelines, a system template as well as recommendations of suitable technologies. The conceptual system framework was used to design a prototype intervention for children with ASD, to improve Theory of Mind through an AR based role play (aka socio-dramatic play) activity. The intervention uses the unique affordances of AR, which allow the user to control a digital character using their own facial expressions to facilitate role play activities. The intervention progressed through a rapid prototyping process where user feedback was gained through UCD methods, including focus group and usability testing. The user centred design approach was then adopted to ascertain and validate the design of the application and the user requirements through two usability studies. The final design is an iPad based AR application using facial expression tracking to interact with the activity. The digital intervention is designed to be used by a child and a guiding adult as a tool to extend and encourage role play. The digital intervention activity is based on familiar characters set in the context of a story book from the Oxford Reading Tree scheme, however, any character-based story book or media could be used. The child uses this to "pretend to be" a character in the story and accurately present their character's emotions at different points in the narrative. The impact of the application on Theory of Mind and overall engagement was evaluated through two sets of case studies, with a total of eight children in two different schools in the North West of the UK. The results establish that an AR face driven interface supports the development of Theory of Mind in children with ASD. The study demonstrated strong evidence for improvement of Theory of Mind for the participants of the study. Engagement in the activity across all cases was very high. This work could lead to interesting future studies related to AR technologies for children with ASD as increased engagement could lead to improved learning and outcomes. The two contributions of this study are, firstly, the user requirements and conceptual system framework generated through a UCD approach which could be used by researchers in the design of digital pretend play interventions for children with ASD. Secondly, the creation of an AR intervention which shows strong support for the development of Theory of Mind for children with ASD. Future research and larger scale studies are recommended to validate the results across a wider group of children with ASD and to determine which subsets of children with ASD would benefit the most from this intervention. The intervention should be studied further for its potential to improve social conversation in children with ASD.

Over the years, a set of story-telling conventions has been established for Television (TV), between content creators and their audiences. These conventions typically involve the creation, and subsequent consumption, of 2-dimensional (2-D) imagery accompanied by sound, requiring limited, if any, interaction between the viewers and content. As consumer Augmented Reality (AR) devices become a commodity, researchers and broadcasters are investigating the ways in which AR can be used to enhance/transform a conventional TV viewing experience - a concept referred to as Augmented Reality Television (ARTV). However, the addition of AR to TV has the potential to disrupt the conventional TV viewing assumptions, since AR's key characteristics are arguably its ability to deliver interactive 3-dimensional (3-D) imagery that is registered to the physical space; a type of content that TV viewers are likely not used to consuming. This raises the question of how to create ARTV content? This thesis conceptualises ARTV as an emerging mass medium, where content creators use both AR and TV to tell their stories to their audiences. We start by providing a few preliminary working definitions (e.g., TV, AR, and ARTV), then systematically review prior research and identify six major themes and six prominent design dimensions (namely, abstraction, interaction, time, display, context, and editorial control). These design dimensions are then used as a basis to create an ARTV design space, from which we derive a cheat sheet to support practitioners. We subsequently evaluate the intelligibility of this cheat sheet, and its impact on conceptualising novel ARTV experiences, in a user study with n = 10 participants. Our findings indicate that the cheat sheet can be useful in conceptualising novel ARTV scenarios, and prompt a series of refinements to the design space and associated cheat sheet. Given the dominance of ARTV to enhance conventional TV viewing experiences that take place in the living room, we further develop this aspect with its own conceptual framework and classification. To demonstrate the applicability of our design space and conceptual framework, we explore two specific types of living room based ARTV experiences in two user studies. In the first, we add holograms to an existing nature documentary and invite n = 12 participants to watch a six-minute ARTV content - where TV content is presented on a physical TV display, and AR content is displayed on a Microsoft Hololens head-mounted display - in a user testing facility that is designed to look and feel like a conventional living room. Our findings suggest that the viewers are likely to find such an experience engaging. However, some may feel a degree of anxiety and a fear of missing parts of the content and underlying narrative. In the second study (n = 10), we use points on the display dimension of our design space to vary presentation of a lifelike programme-related hologram as part of a nature documentary show. We explore six points on the display dimension in the context of an ARTV experience where both the TV and the living room are virtual. We investigate the impact of the hologram's movement behaviour on the viewers. Our findings highlight the role of personal preferences and the perceived role of the hologram in the underlying narrative and TV content. Overall, the thesis highlights a need to carefully consider the impact of additional augmentations on the viewers. Furthermore, viewers should be given the ability to customise their ARTV experiences, based on personal preferences. Our contributions are the following: 1) a systematic literature review, highlighting six common themes of research in ARTV and six commonly used design dimensions; 2) creation and evaluation of a design space for ARTV and its subsequent refinement based on the evidence gathered from a user study; 3) a conceptual framework for ARTV experiences in the living room, identifying nine types of ARTV and highlighting areas for future research; 4) two user-studies investigating two types of ARTV, highlighting a need to accommodate personal preferences for ARTV content.

This portfolio of compositions uses novel biometrics from gestural interfaces called Myo armbands to compose five pieces of electroacoustic music within a continuum of spatial environments, ranging from the real to the virtual. The portfolio achieves this by processing biometric data from the Myo armbands within developed software (using Max 8), applying machine learning to such data (via Wekinator) and mapping predictive outputs to audiovisual materials across the spatial continuum. The results of this approach show how unique compositional affordances can be created when using biometrics within different spatial environments and how implementing machine learning methods can help to create such opportunities. The outcomes are presented via five composed portfolio works, three published papers within the scientific community, and numerous music performances navigating the overlapping areas of music composition and computer science. By using novel biometrics to compose music within a continuum of space, this interdisciplinary research topic contributes original knowledge to both the fields of music composition and computer science (human-computer interaction).

Dementia affects more than 850,000 people in the UK, and this figure continues to rise year on year. People living with dementia often have difficulties completing activities of daily living (ADLs), leading to a reliance on family or professional carers. However, assistive technology, such as task prompting tools, can support people living with dementia to maintain their independence and live at home for longer. Augmented Reality (AR) is an increasingly prevalent technology and has been used as a task prompting tool in industrial settings to support complex maintenance and assembly tasks. The use of AR for task assistance has also been identified as a promising area for development in Domestic AR. Despite the use of AR as a task prompting tool in industrial settings and the potential of AR in domestic settings, relatively little is known about the efficacy of augmentations for ADLs. The work in this thesis aims to provide an initial exploration into the use of AR as a task prompting tool to support people living with dementia to complete ADLs at home. Multiple stakeholders, including health professionals, a general adult population, older adults without cognitive impairment, and people living with dementia and their family carers, have been included to develop a holistic understanding of the use of AR in a domestic setting, based on four studies carried out for this project. The first study consisted of in-person interviews with professionals with experience of working with people living with dementia. The second study was a lab experiment with older adults to compare four AR visual prompting techniques to prompt five basic actions found in many ADLs. The third study involved the co-design of AR prompts in a kitchen context, and their evaluation with a general adult population using an online survey. The final study consisted of online interviews with people living with dementia and their family carers to explore the results of the previous three studies and how AR could be beneficial from the point of view of people with lived experience of dementia. The overall findings show that AR as a tool to support task prompting of domestic tasks was received positively by the participants of these studies. A combination of text, audio, and a ghosthand image demonstrating the action to carry out could be most beneficial for people with dementia, but AR prompts should be easily customisable to cater for different abilities, preferences, and personalities. Furthermore, early introduction of AR will be key for uptake when the technology has been developed further. The potential of domestic AR to improve the lives of people affected by dementia and those that support them with ADLs is considered as motivation for future work in this promising research area.

This thesis presents an analysis of the implementation of Spatial Augmented Reality (SAR) to support collaborative design. SAR is a class of Augmented Reality (AR), where the AR effect is independent from the viewer's perspective. This property should make SAR uniquely well suited to support collaborative design sessions as it allows multiple participants to simultaneously view the same AR enhanced models concurrently. However, the adoption of SAR in the field of design remains limited. Thus, there is a knowledge gap that surrounds SAR, due to the novelty of the technology, and how its implementation can be configured to better support design. The aim of this research is therefore to: "Investigate SAR Systems' Characteristics and Features and how these Affect the Efficacy of Co-Design Sessions". A review of the literature surrounding collaborative design is provided to highlight evidence that enhanced collaboration during design sessions results in improved outcomes. Thereafter AR, and SAR in particular, are analysed to provide a basis for the continued investigation of SAR as a valid tool to support co-design sessions. In addition to the review, the state-of-the-art in implementing SAR in design and collaborative design is presented and discussed to frame the subsequent research conducted as part of this PhD. Moreover, literature is provided covering the metrics that will be used to analyse SAR supported co-design sessions. The methodology implemented throughout this thesis was one of empirical research conducted through a number of experimental studies aimed at evaluating the impact of SAR on co-design sessions. The experimental nature of this research thus required the development of an SAR platform to be used during experimental testing. The platform was developed iteratively and improved on the basis of feedback received from experiments. These experiments focused on three main areas: 1. The impact of SAR on co-design sessions as a whole. These experiments aid in understanding the impact that SAR has on the overall design session, focusing primarily on the design outcomes. 2. The analysis of specific characteristics and features of SAR and their impact on design outcomes. Experiments used to analyse the impact of specific elements that are innate to SAR to evaluate their individual impact on SAR supported design sessions. 3.Experiments and data collection aimed at gathering industry feedback. Used to understand the industry perspective to allow for the optimization of SAR to cater to the needs of industry and ultimately allow for its wider adoption. The results highlighted the viability of SAR as a tool to support collaborative design sessions. However, the studies also identified a number of areas for improvement in the implementation of SAR. The thesis concludes with a summary of thirty-two findings, condensing them into two lists of recommendations. One of these lists for technical improvements and one for features and affordances, aimed at assisting future researchers wishing to develop improved SAR platforms for co-design.

Vision loss is rarely absolute; most people referred to as blind have some form of residual vision. This residual vision can create a confusing view of the world. In this thesis, I explore how a camera system and a wearable display can augment an image of the word to improve its visibility and understanding to a visually impaired wearer. I focus on two crucial parts of life; recognising faces and reading. In the first part of this thesis, I design and validate a new face perception test to assess different image enhancement forms. I used this new test to measure the impact of image enhancement on a visually impaired population (n=64). The results found that approximately 40% of participants showed a significant improvement in test scores. Performance was related to reading acuity and lower spatial frequency loss. I then develop and integrate a complete real-time image processing solution for face perception enhancement with an augmented reality display, demonstrating a practical application of the image enhancements. The second part of this thesis's focus is reading. I compare paper-based reading performance against reading with a virtual reality (VR) display with a visually impaired population (n=53). I further assess the impact of different colours on reading with the VR display. The results found no difference in reading performance, comparing VR to paper- based. The colour of text and background had a significant effect, with 75% of participants having a preferred colour combination. A participant with a higher overall magnitude of colour vision loss is more likely to benefit from white text on a black background than black text on a yellow background. Augmenting an individual's residual vision can improve the quality of information they receive about the visual world. While not universal, this improvement has the potential to enhance a visually impaired person's quality of life significantly.

This study aims to investigate the effects of augmented reality activities in both traditional and flipped learning method classroom environments and compare the efficacy of these activities in the context of English phrasal verbs. For this purpose, the study was carried out within the scope of a college of foreign language English language preparation course, and the students' academic achievements, motivation and use of learning strategies were examined. In addition, the opinions of the course lecturer and students on augmented reality, augmented reality activity experiences, and expectations of the future of augmented reality were all examined within the scope of the research. A mixed-method approach was used with a quasi-experimental research design in the study, which includes a pre-test, a post-test, and a control group. The research study group consists of 61 students from the College of Foreign Languages' English preparatory course. The research was carried out within the English language preparatory course scope in the context of phrasal verbs in the fall semester of the 2019-2020 academic year. There were two experimental research groups, one of which followed the traditional classroom instruction plan, and the other followed the flipped learning method instruction plan. The control group followed the standard curriculum, using existing classroom technology. Both experimental research groups performed the same augmented reality activities throughout the research process. Descriptive statistics, one-way Analysis of Variance (ANOVA), repeated measure variance analysis, and the Kruskal-Wallis test were all used to analyse the study's quantitative data. In addition, descriptive analysis was performed on the qualitative data. As shown by the study results, the students' academic achievement scores in the experimental groups increased more than the control group students, who only followed the curriculum. However, at the end of the experimental process, it was concluded that there were no significant differences between the students' pre-test and post-test motivational belief and learning strategy scores. Furthermore, when the findings obtained from the semi-structured interviews were examined, the course lecturer and students from the experimental research groups generally stated that augmented reality activities could increase lesson success and motivation. Lastly, several suggestions were made based on the results of the study.

It is widely claimed that creativity is becoming increasingly important as Artificial Intelligence takes over more and more administrative roles. According to the OECD, many employers now consider creativity a key skill for success in the workplace and this has been highlighted in recent OECD papers focusing on key competencies of 2030. With advancements of new technologies such as augmented reality (AR), new technological affordances have the capabilities of supporting collaborative creative-processes in new and exciting ways. It is believed that no prior research has focused on AR and teaching for creativity - this research aims to do just that. This thesis employs a design-based research methodology, aiming to understand how a 3D AR intervention can be designed, tested and evaluated to support collaborative creativity in an art and design context at lower secondary school level. Relevant literature pertaining to creativity is reviewed and a multi-level conceptualisation of creativity that encompasses individual and group-level processes is proposed as a way to guide the research. A review of the field of AR for educational purposes is also conducted and findings suggest that a) more research is necessary in the field of art and design and b) the field lacks appropriate application of methods that identify the processes underpinning observed outcomes, such as increased motivation and academic performance. Both of these limitations are aimed to be addressed by this study through the development and documentation of an open-ended AR 3D modelling creativity task. Conjecture mapping is used as a method to outline how specific AR affordances relating to the application of interest (Grib) lead to hypothetical learning trajectories derived from theory. These are developed over three iterative cycles of testing and refinement that included 84 participants in Cycle 1, 8 of the same participants in Cycle 2, and 11 of the same participants in Cycle 3. Participants were aged 12- 14 years and from a secondary school based in Gibraltar. Data were collected from video and audio recordings of group interactions, as well as pre and post- questionnaires pertaining to various psychological factors relevant to creativity. A sociocultural perspective is taken for the analysis of group interactions and a dialogic framework of creativity that includes nonverbal interactions is used in an attempt to uncover a holistic understanding of all modes of communication. Epistemic Network Analysis (ENA) and measures of factors such as motivation, risk- taking and creative self-concept beliefs are used in an effort to uncover relationships between constructs in more detail. Analysis of the multimodal data revealed that the intervention design - which included the AR application 'Grib' - effectively helped students to engage in important individual and group creativity- relevant processes like divergent thinking, convergent thinking, intrinsic motivation and social cohesiveness. Several key affordances of the Grib application were identified as helping students engage in these processes and included, textual layering, sketch-to-3D, collaborative functonality and inspection of objects in 3D. Teaching practices identified as supporting students' creativity-relevant processes included time management, encouragement/reinforcement, reflection of the process, and social cohesion. Based on empirical findings from the study, design principles aimed at practitioners, researchers and designers seeking to support collaborative creative thinking in lower secondary school students are proposed. The study also contributed to a more holistic understanding of creativity by proposing a conceptual framework that drew on and extended numerous models of creativity. It offers a more micro understanding of specific factors that may be responsible for individual and group processes during creative stages. Practical design principles for future practitioners and researchers are also proposed. The above contributions are anticipated to be of practical and theoretical significance to those interested in understanding how individual and group processes might be assembled during similar creative tasks.

Augmented Reality (AR) technology facilitates the learning of 3D geometry because it allows students to easily manipulate 3D geometric shapes. Besides, gamification can further enhance the potential of AR technology as an educational tool by increasing user engagement. However, some methodological limitations were identified in the existing work on this specific topic. First, one of the major drawbacks was the lack of control groups for evaluating the effect of learning geometry with the conventional teaching methods as compared to that of the AR-based approaches. Second, the AR application assessment focused on learning outcomes, neglecting the user experience aspect. Therefore, the main research question of this PhD research project was to explore to what extent AR-based applications with gamification could make learning 3D geometry more effective and engaging for students as compared with the traditional 2D approach. To address this question, two empirical studies were conducted. In Study 1, an AR application for learning cross-sectional shapes and variables in 3D geometry was developed. The AR-based approach was compared with the traditional chalk-and-board one by involving sixty 12-16 year-olds from two schools in Thailand. In Study 2, how certain gamification elements - Badges, Points and Timer - influenced the motivational effect of the AR application in the learning of 3D geometric shapes was examined. It involved 150 students of 12-16 years old. A motivation questionnaire for measuring Interest, Confidence and Intention to Use level was constructed. Results indicated that the AR classes showed significantly stronger learning effects than the traditional ones, especially for more complex geometric concepts. However, there were no statistically significant differences in the motivational effect between the gamified and non-gamified AR applications, and Points had a stronger learning effect than Badges or Timer. One implication for designing gamified AR educational applications is that time pressure seems an inappropriate gamification element.

Real-time solutions to reducing the gap between virtual and physical worlds for photorealistic interactive Augmented Reality (AR) are presented. First, a method of texture deformation with image inpainting, provides a proof of concept to convincingly re-animate fixed physical objects through digital displays with seamless visual appearance. This, in combination with novel methods for image-based retargeting of real shadows to deformed virtual poses and environment illumination estimation using in conspicuous flat Fresnel lenses, brings real-world props to life in compelling, practical ways. Live AR animation capability provides the key basis for interactive facial performance capture driven deformation of real-world physical facial props. Therefore, Intermediated Reality (IR) is enabled; a tele-present AR framework that drives mediated communication and collaboration for multiple users through the remote possession of toys brought to life. This IR framework provides the foundation of prototype applications in physical avatar chat communication, stop-motion animation movie production, and immersive video games. Specifically, a new approach to reduce the number of physical configurations needed for a stop-motion animation movie by generating the in-between frames digitally in AR is demonstrated. AR-generated frames preserve its natural appearance and achieve smooth transitions between real-world keyframes and digitally generated in-betweens. Finally, the methods integrate across the entire Reality-Virtuality Continuum to target new game experiences called Multi-Reality games. This gaming experience makes an evolutionary step toward the convergence of real and virtual game characters for visceral digital experiences.

Augmented reality (AR) is the enhancement of our perception of the world around us. It promises the ability to simultaneously interact with a virtual environment and real objects, providing an intuitive method to interact with data. However, AR displays must be lightweight and compact, and be able to display bright, high-resolution 3D images. Current AR displays are limited by bulky-form factors with a limited field of view (FOV), and the inability to display high quality 3D images without introducing side effects such as nausea and eyestrain. In the first part of this thesis an afocal Maxwellian display is analysed and modelled, the small eyebox limitation examined and a solution proposed. A prototype head mounted display unit was constructed as a demonstration. The display performance was analysed objectively and subjectively, and found to perform well over a wide accommodation range. However, the system is mechanically complicated to provide for a large IPD demographic, and the implementation of the eyebox expansion is too limited. An alternative curved reflector system was proposed to reduce weight, increase FOV, and increase the eyebox. The final part of this research examines waveguides as an AR display technology. The limitations of current systems are evaluated, and a novel switchable waveguide architecture proposed where light can be selectively directed towards the eye to form an image time sequentially. This enables higher quality images, with a wider FOV and increased efficiency, from a more transparent waveguide. A liquid crystal diffraction grating is developed as a switching element with a simulated diffraction efficiency of 20% and a geometry optimised for easy industrial manufacture. The possibility for such a system to include an integrated lens function is also considered, and an 80° FOV design presented.

The central hypothesis of this thesis is that probabilistic user interface design provides an effective methodology for delivering productive and enjoyable applications in virtual reality (VR) and augmented reality (AR). This investigation is timely given the recent emergence of mass-market virtual and augmented reality head-mounted displays and growing demand for tailored applications and content. The design guidance for building compelling and productive applications for these environments is, however, currently lagging the pace at which the underlying technology is maturing. This is problematic given important differences between designing conventional 2D interfaces and interactions and their embodied 3D counterparts. This dissertation investigates probabilistic user interface design as a method for solving many of the novel challenges encountered when developing applications for VR and AR. Probabilistic user interface design seeks to model the uncertain events in a system and identify, implement and validate strategies that drive improved system performance. This thesis addresses four research questions by applying a probabilistic treatment in four distinct but closely related case studies. These four case studies are selected to illustrate the flexibility and unique benefits offered by this method. Research Question 1 asks how the probabilistic qualities of an interface can be determined and how this can inform design. This question is investigated in the context of text entry in VR with a probabilistic characterisation performed on two fundamental design choices. Research Question 2 relates to the challenge of adapting AR applications to deployment contexts not knowable at design time. A study in which crowdworkers are employed to build a probabilistic understanding of the requirements for contextually adaptive AR answers this question. The text entry theme is revisited in answering Research Questions 3 which asks how high levels of input noise can be mitigated through inference. A probabilistic text entry method specifically tailored for use in AR is implemented and evaluated. Finally, Research Question 4 asks how the high dimensional design space in AR and VR applications can be efficiently explored to support ideal design choices. Interface refinement through probabilistic optimisation and crowdsourcing is shown to be highly efficient and effective for this purpose. A probabilistic treatment in the design process has many potential benefits, principle among which is increased robustness to circumstances unanticipated at design time. This thesis contributes to the toolset and guidance available to designers and supports the development of next generation user interfaces specifically tailored to virtual and augmented reality.

Technicians' performance is a major driver in maintenance and each process can be prone to time and quality variances as well as errors due to factors such as experience, complexity and environment. Augmented Reality (AR) is an emerging technology that has been applied in a wide variety of disciplines and has been demonstrated to have a role with improving efficiency, effectiveness and decision-making within industrial maintenance. AR has not reached its full potential yet and its implementation in Industry is slowed down by three main limitations: hardware restricted capabilities, object recognition robustness and contents-related issues. This PhD project focuses on easing the implementation of AR by overcoming the AR technology selection challenges and the AR contents-related issues. In order to reach the aim, the student has provided three main contributions to knowledge: 1) a process to select AR technology for maintenance (IPSAR), 2) a method for creating AR step-by-step procedures (FARP) and 3) a method for providing remote assistance (ARRA). FARP and ARRA methods have been developed and tested. The first allows recording procedures in an ad-hoc designed "AR-format" and is able to show "step-by-step" procedures. It aims to support deskilling the maintenance process and reducing the error rate by simplifying the delivery of maintenance with efficient and effective guidance. The second overcomes current remote video-call assistance limitations by improving spatial referencing. ARRA module allows to provide AR-assistance by overlaying virtual objects on the real environment of a remote maintainer. The methods proposed by the student could boost the implementation of AR and open the doors for a bright future in which AR supports technicians thus reducing operational costs and training and improving human performances.

We have explored applications of Augmented Reality (AR) for external beam radiotherapy to assist with treatment planning, patient education, and treatment delivery. We created an AR development framework for applications in radiotherapy (RADiotherapy Augmented Reality, RAD-AR) for AR ready consumer electronics such as tablet computers and head mounted devices (HMD). We implemented in RAD-AR three tools to assist radiotherapy practitioners with: treatment plans evaluation, patient pre-treatment information/education, and treatment delivery. We estimated accuracy and precision of the patient setup tool and the underlying self-tracking technology, and fidelity of AR content geometric representation, on the Apple iPad tablet computer and the Microsoft HoloLens HMD. Results showed that the technology could already be applied for detection of large treatment setup errors, and could become applicable to other aspects of treatment delivery subject to technological improvements that can be expected in the near future. We performed user feedback studies of the patient education and the plan evaluation tools. Results indicated an overall positive user evaluation of AR technology compared to conventional tools for the radiotherapy elements implemented. We conclude that AR will become a useful tool in radiotherapy bringing real benefits for both clinicians and patients, contributing to successful treatment outcomes.

This thesis presents a real-time automated building progress monitoring solution for indoor environments using a mobile device. Such a system could prompt accurate and timely assessment of work progress that would allow managers to make adjustments and minimise both time and cost overrun when deviations from the schedule occur. Although many researchers have proposed approaches for progress monitoring in outdoor scenes, these cannot perform in real-time and shift into the complex interior environment. Research efforts for indoor environments are not fully automated and lead to errors in more complex scenes. Systems based on mobile devices could potentially enhance the inspection process and reduce the required time by allowing the inspector to acquire progress data by simply walking around the site. The main challenge of these systems is the tracking of the pose of the camera to achieve accurate alignment between the 3D design model and the real-world scene. Methods for estimating the user’s pose rely on a) tags on each target of interest, which require additional time and cost for installation and maintenance; b) pre-selected user locations, which restricts the user to those locations only; or c) GPS on the augmented reality headset, which only applies to outdoor inspections. Additionally, current mobile-based inspection systems do not perform any comparison between the captured as-built and the as-planned data. In this research, different marker-less Augmented Reality (AR) potential methods were implemented and tested for finding the most robust tracking solution. The Microsoft HoloLens was found to be the top performer for tracking the user’s pose and for overall user-experience. Next, a semi-automated method was developed for initially registering the 3D model to the real environment by exploiting information from detected floor and wall surfaces. Results showed that this method reduces the time of the initial registration by 58%. Having the 3D model aligned to the real environment and knowing the pose of the camera at every moment, an automated method was developed that exploits the captured as-built surface mesh data from the mobile device, compares it against the 3D design model and identifies in real-time whether an object has been built according to plan. Different parameters were tested for finding the optimum combination based on the current quality of mesh data. If quality of mesh data changes, then new parameters should be explored. Finally, the proposed solution was tested in real site conditions resulting in 76.6% precision, 100.0% recall, and 83.5% accuracy.

Augmented Reality (AR) is a recent technology that allows a seamless composition between virtual objects and the real world. This practice-based research uses the affordances of AR to design an AR textbook for collaborative learning experience. It identifies the key concepts of children s AR textbooks for the designing and evaluation of collaborative learning experiences. These concepts were used to develop a conceptual framework for the AR textbook that considers collaborative experience, learning and usability. Informed by these concepts, the research also has identified the design features which are unique to AR affordances which can be integrated in the school textbooks to develop a collaborative AR textbook for primary school children. The research follows a participatory design approach to involve the users of the AR textbook in the design process. The researcher has conducted three co-design studies involving primary school children and adults using cooperative inquiry techniques. The first study uses low-tech prototyping to find the overall direction of designing the AR textbook. After the development of the first AR textbook prototype, two formative evaluations have been conducted using cooperative inquiry critiquing, and layered elaboration techniques. Throughout these studies, a conceptual framework has been developed namely, Experience, Learn and Use (ELU) for the designing and evaluation of children s AR textbooks for collaborative learning experience. This framework is based on the adaption of Janet Read s Play, Learn, Use (PLU) model that defines children s relationships with the interactive technologies. The research proposes the ELU framework as a useful classification framework in the evaluation process, which informs the design features of the AR textbook which are related to the concepts of collaborative experience, learning and usability. The practical component of the thesis proposes a sample of an AR textbook that is integrated in the regular school curriculum. It demonstrates the design features which can be implemented in other textbooks to support collaborative learning experiences for primary school children. The documentation of the co-design process provides a practical framework for co-designing an AR textbook with children, as well as an evidence of using the ELU framework in practice. 4 This research also contributes in bridging the gap between AR and Child-Computer Interaction (CCI) communities, through the use of common CCI methods in the AR development. This research has resulted in key design principles which contribute original knowledge to the literature of the AR for children s education considering the CCI perspective. These important principles are informed by the collaborative experiences, learning and usability aspects that establish a framework for the design and evaluation of collaborative AR textbook for children. The eight identified principles by this research are, Joint Textbooks, Personalised AR Experience, Interactive AR Book, Communication-Based Learning, Rewarding AR feedback, Audio AR Textbook, Intuitive AR Markers, and Mutual AR Display. The research introduces the definition for each of the concepts and a demonstration of the related design features in the outcome of the AR textbook prototype.

Double Vision: A practice- led investigation of art and differential perception is a series of five interrelated practice-led research studies into artistic expression controlling perceptual experiences between audiences of varying visual acuities. Significant refinements  occurred between the first and second, and second and third studies. The last four studies were conducted with the aim of understanding vision’s influence on perception. Double Vision’s lead methodological approach was artistic practice. Other methods were employed according to the needs of that practice. They included iteration, collaboration, exhibition and testing. The research questions of Double Vision were refined in response to the results of artistic practice. That evolution resulted in two interrelated questions: Can artwork be intentionally created to be experienced differently dependent on one’s visual abilities? and If so, can those experiences be shared? A further question, ‘Can an analogy to colour deficient vision be created that engages both those with colour vision deficiency and the typically sighted?’, concludes the investigations. Artwork was realized through printmaking, animation and multimedia formats. Its context and content derived from many forms, notably the Ishihara Test for Colour Deficiency, writings of William Blake, contemporary music and philosophy. Augmented reality was employed to facilitate the translation of visual perceptions between targeted audiences. A number of exhibitions were held exploring these themes.

Alternate realities have fascinated mankind since early prehistory and with the advent of the computer and the smartphone we have seen the rise of many different categories of alternate reality that seek to augment, diminish, mix with or ultimately replace our familiar real world in order to expand our capabilities and our understanding. This thesis presents parallel reality as a new category of alternate reality which further addresses the vacancy problem that manifests in many previous alternate reality experiences. Parallel reality describes systems comprising two environments that the user may freely switch between, one real and the other virtual, both complete unto themselves. Parallel reality is framed within the larger ecosystem of previously explored alternate realities through a thorough review of existing categorisation techniques and taxonomies, leading to the introduction of the combined Milgram/Waterworth model and an extended definition of the vacancy problem for better visualising experience in alternate reality systems. Investigation into whether an existing state of the art alternate reality modality (Situated Simulations) could allow for parallel reality investigation via the Virtual Time Windows project was followed by the development of a bespoke parallel reality platform called Mirrorshades, which combined the modern virtual reality hardware of the Oculus Rift with the novel indoor positioning system of IndoorAtlas. Users were thereby granted the ability to walk through their real environment and to at any point switch their view to the equivalent vantage point within an immersive virtual environment. The benefits that such a system provides by granting users the ability to mitigate the effects of the extended vacancy problem and explore parallel real and virtual environments in tandem was experimentally shown through application to a use case within the realm of cultural heritage at a 15th century chapel. Evaluation of these user studies lead to the establishment of a number of best practice recommendations for future parallel reality endeavours.

Learning is one of the most interactive processes that humans practice. The level of interaction between the instructor and his or her audience has the greatest effect on the output of the learning process. Recent years have witnessed the introduction of e-learning (electronic learning), which was then followed by m-learning (mobile learning). While researchers have studied e-learning and m-learning to devise a framework that can be followed to provide the best possible output of the learning process, m-learning is still being studied in the shadow of e-learning. Such an approach might be valid to a limited extent, since both aims to provide educational material over electronic channels. However, m-learning has more space for user interaction because of the nature of the devices and their capabilities. The objective of this work is to devise a framework that utilises augmented reality and context awareness in m-learning systems to increase their level of interaction and, hence, their usability. The proposed framework was implemented and deployed over an iPhone device. The implementation focused on a specific course. Its material represented the use of augmented reality and the flow of the material utilised context awareness. Furthermore, a software prototype application for smart phones, to assess usability issues of m-learning applications, was designed and implemented. This prototype application was developed using the Java language and the Android software development kit, so that the recommended guidelines of the proposed framework were maintained. A questionnaire survey was conducted at the University, with approximately twenty-four undergraduate computer science students. Twenty-four identical smart phones were used to evaluate the developed prototype, in terms of ease of use, ease of navigating the application content, user satisfaction, attractiveness and learnability. Several validation tests were conducted on the proposed augmented reality m-learning verses m-learning. Generally, the respondents rated m-learning with augmented reality as superior to m-learning alone.

This research explores how mobile augmented reality experiences are perceived and interacted with by using a quantitative eye-tracking study to record user eye movement behavior. Visual communication is vital in today’s increasingly visual society and current technology gives evidence of much promise in adding additional engagement, information, and interactivity through augmented reality. Though smartphone technology allows all smartphone users to access and experience browser-based augmented reality, there is little research on design aspects from a user experience perspective, such as the influence of layered depths in an augmented scene. The purpose of this study was to explore the variable of augmented reality depths and their impact on user engagement and interest within the framework of perception theory. A quantitative research method of eye tracking was used to measure fixations, and a user experience Likert-style question was used to assess the level of interest between single-layer and multiple-layer augmented reality experiences. The results found that total gaze time was increased for one of the multilayered stimuli, and participants ranked the multilayered experiences as more interesting than the single layer. This study has theoretical implications within the visual communication perception theory and methodological implications for eye tracking and user experience. This study also has practical implications for content creators that seek to enhance visual communication engagement and interest through augmented reality and suggestions are offered for future research.

Augmented reality (AR) is seeing a rapid expansion into several domains due to the proliferation of more accessible and powerful hardware. While augmented reality user interfaces (AR UIs) allow the presentation of information atop the real world, this extra visual data potentially comes at a cost of increasing the visual clutter of the users' field of view, which can increase visual search time, error rates, and have an overall negative effect on performance. Visual clutter has been studied for existing display technologies, but there are no established measures of visual clutter for AR UIs which precludes the study of the effects of clutter on performance in AR UIs. The first objective of this research is to determine the applicability of extant image analysis measures of feature congestion, edge density, and sub-band entropy for measuring visual clutter in the head-worn optical see-through AR space and establish a relationship between image analysis measures of clutter and visual search time. These image analysis measures are specifically chosen to quantify clutter, as they can be applied to complex and naturalistic scenes, as is common to experience while using an optical see-through AR UI. The second objective is to examine the effects of AR UIs comprised of multiple apparent depths on user performance through the metric of visual search time. The third objective is to determine the effects of other AR UI properties such as target clutter, target eccentricity, target apparent depth and target total distance on performance as measured through visual search time. These results will then be used to develop a visual clutter score, which will rate different AR UIs against each other. Image analysis measures for clutter of feature congestion, edge density, and sub-band entropy of clutter were correlated to visual search time when they were taken for the overall AR UI and when they were taken for a target object that a participant was searching for. In the case of an AR UI comprised of both projected and AR parts, image analysis measures were not correlated to visual search time for the constituent AR UI parts (projected or AR) but were still correlated to the overall AR UI clutter. Target eccentricity also had an effect on visual search time, while target apparent depth and target total distance from center did not. Target type and AR object percentage also had an effect on visual search time. These results were synthesized into a general model known as the "AR UI Visual Clutter Score Algorithm" using a multiple regression. This model can be used to compare different AR UIs to each other in order to identify the AR UI that is projected to have lower target visual search times.

The creation of an effective construction schedule is fundamental to the successful completion of a construction project. Effectively communicating the temporal and spatial details of this schedule are vital, however current planning approaches often lead to multiple or misinterpretations of the schedule throughout the planning team. Four Dimensional Computer Aided Design (4D CAD) has emerged over the last twenty years as an effective tool during construction project planning. In recent years Building Information Modelling (BIM) has emerged as a valuable approach to construction informatics throughout the whole lifecycle of a building. Additionally, emerging trends in location-aware and wearable computing provide a future potential for untethered, contextual visualisation and data delivery away from the office. The purpose of this study was to develop a novel computer-based approach, to facilitate on-site 4D construction planning through interaction with a 3D construction model and corresponding building information data in outdoor Augmented Reality (AR). Based on a wide ranging literature review, a conceptual framework was put forward to represent software development requirements to support the sequencing of construction tasks in AR. Based on this framework, an approach was developed that represented the main processes required to plan a construction sequence using an onsite model based 4D methodology. Using this proposed approach, a prototype software tool was developed, 4DAR. The implemented tool facilitated the mapping of elements within an interactive 3D model with corresponding BIM data objects to provide an interface for two way communication with the underlying Industry Foundation Class (IFC) data model. Positioning data from RTK-GPS and an electronic compass enabled the geo-located 3D model to be registered in world coordinates and visualised using a head mounted display fitted with a ii forward facing video camera. The scheduling of construction tasks was achieved using a novel interactive technique that negated the need for a previous construction schedule to be input into the system. The resulting 4D simulation can be viewed at any time during the scheduling process, facilitating an iterative approach to project planning to be adopted. Furthermore, employing the IFC file as a central read/write repository for schedule data reduces the amount of disparate documentation and centralises the storage of schedule information, while improving communication and facilitating collaborative working practices within a project planning team. Post graduate students and construction professionals evaluated the implemented prototype tool to test its usefulness for construction planning requirements. It emerged from the evaluation sessions that the implemented tool had achieved the essential requirements highlighted in the conceptual framework and proposed approach. Furthermore, the evaluators expressed that the implemented software and proposed novel approach to construction planning had potential to assist with the planning process for both experienced and inexperienced construction planners. The following contributions to knowledge have been made by this study in the areas of 4D CAD, construction applications of augmented reality and Building Information Modelling; · 4D Construction Planning in Outdoor Augmented Reality (AR) · The development of a novel 4D planning approach through decomposition · The deployment of Industry Foundation Classes (IFC) in AR · Leveraging IFC files for centralised data management within real time planning and visualisation environment.

With the advent of ever smaller and more powerful portable computing devices, and ever smaller cameras, wearable computing is becoming more feasible. The ever increasing numbers of augmented reality applications are allowing users to view additional data about their world overlaid on their world using portable computing devices. The main aim of this research is to enable a user of a wearable robot to explore large environments automatically viewing augmented reality at locations and on objects of interest. To implement this research a wearable visual robotic assistant is designed and constructed. Evaluation of the different technologies results in a final design that combines a shoulder mounted self stabilizing active camera, and a hand held magic lens into a single portable system. To enable the wearable assistant to locate known objects, a system is designed that combines an established method for appearance-based recognition with one for simultaneous localization and mapping using a single camera. As well as identifying planar objects, the objects are located relative to the camera in 3D by computing the image-to-database homography. The 3D positions of the objects are then used as additional measurements in the SLAM process, which routinely uses other point features to acquire and maintain a map of the surroundings, irrespective of whether objects are present or not. The monocular SLAM system is then replaced with a new method for building maps and tracking. Instead of tracking and mapping in a linear frame-rate driven manner, this adopted method separates the mapping from the tracking. This allows higher density maps to be constructed, and provides more robust tracking. The flexible framework provided by this method is extended to support multiple independent cameras, and multiple independent maps, allowing the user of the wearable two-camera robot to escape the confines of the desk top and explore arbitrarily sized environments. The final part of the work brings together the parallel tracking and multiple mapping system with the recognition and localization of planar objects from a database. The method is able to build multiple feature rich maps of the world and simultaneously recognize, reconstruct and localize objects within these maps. The object reconstruction process uses the spatially separated keyframes from the tracking and mapping processes to recognize and localize known objects in the world. These are then used for augmented reality overlays related to the objects.

As the frontiers and futures of work evolve, humans and machines will begin to share a more cooperative working space where collaboration occurs freely amongst the constituent members. To this end, it is then necessary to determine how information should flow amongst team members to allow for the efficient sharing and accurate interpretation of information between humans and machines. Shared situation awareness (SSA), the degree to which individuals can access and interpret information from sources other than themselves, is a useful framework from which to build design guidelines for the aforementioned information exchange. In this work, we present initial Augmented/virtual reality (AR/VR) design principles for shared situation awareness that can help designers both (1) design efficacious interfaces based on these fundamental principles, and (2) evaluate the effectiveness of candidate interface designs based on measurement tools we created via a scoping literature review. This work achieves these goals with focused studies that 1) show the importance of SSA in augmented reality-supported tasks, 2) describe design guidelines and measurement tools necessary to support SSA, and 3) validate the guidelines and measurement tools with a targeted user study that employs an SSA-derived AR interface to confirm the guidelines distilled from the literature review.

Maintaining good oral hygiene is crucial for a healthy life, and properly tooth brushing stands out as a fundamental defense against dental issues. Unfortunately, many people do not brush their teeth properly and there is very limited technology available to assist them in compliance with the standard toothbrushing procedure. In the realm of technological innovation, augmented reality stands out as a powerful tool with boundless possibilities. Leveraging the potential of this transformative technology, we aim to utilize mirrors as a unique medium to design a captivating 3D projected model of teeth. This innovative approach aims to inspire and motivate users to elevate their brushing habits, ultimately leading to optimal oral health.

Many people desire to learn a new skill or craft, but having access to a mentor available to provide one on one feedback can be expensive or inaccessible to many. Learning a new skill can bring meaning and added value to a person’s life. Glasses explores bridging the physical and digital world through augmented reality tools that assist in the hand making of products and the learning of a new craft. Though Glasses specifically focuses on glass blowing, the concept could just as easily be applied to any form of craftsmanship such as woodworking, metalsmithing or welding. The product helps inform users of the best way to move in 3D space to create their desired form. Additionally, they can view a model projected onto the material they’re working to better understand and reference the form they’re trying to create. It is easy to use, and relatively hands free featuring a simple, customizable physical interface. Having direct feedback from an expert is often the best way to learn a new skill, Glasses is the next closest thing for those without one. Glasses is a full product line, featuring a physical set of safety glasses and the software to go along with them.

Extended reality, or "XR", is the adopted umbrella term that is heavily gaining traction to collectively describe Virtual reality (VR), Augmented reality (AR), and Mixed reality (MR) technologies. Together, these technologies extend the reality that we experience either by creating a fully immersive experience like in VR or by blending in the virtual and "real" worlds like in AR and MR. The sustained success of XR in the workplace largely hinges on its ability to facilitate efficient user interactions. Similar to interacting with objects in the real world, users in XR typically interact with virtual integrants like objects, menus, windows, and information that convolve together to form the overall experience. Most of these interactions involve near-field object manipulation for which users are generally provisioned with visual representations of themselves also called self-avatars. Representations that involve only the distal entity are called end-effector representations and they shape how users perceive XR experiences. Through a series of investigations, this dissertation evaluates the effects of virtual end effector representations on near-field object retrieval interactions in XR settings. Through studies conducted in virtual, augmented, and mixed reality, implications about the virtual representation of end-effectors are discussed, and inferences are made for the future of near-field interaction in XR to draw upon from. This body of research aids technologists and designers by providing them with details that help in appropriately tailoring the right end effector representation to improve near-field interactions, thereby collectively establishing knowledge that epitomizes the future of interactions in XR.

This thesis project is an attempt to create an integration of augmented reality (AR) technology and tabletop board games, with a focus on tackling the challenge of maintaining engagement and capturing the attention of modern audiences who are often drawn to more interactive forms of entertainment like movies and video games. Specifically, the project explores the concept of augmenting older tabletop games with a combination of futuristic digital graphics and overlays with original physical game elements. The project sees and recognizes the problem, which is that the disparity between the level of engagement offered by traditional tabletop games and the immersive experiences provided by modern digital entertainment is quite large. It identifies the problem of older tabletop games lacking the ability to sustain audience interest, particularly in comparison to their digital counterparts. To bridge this gap, the thesis proposes the use of AR technology, which offers a promising avenue to enhance the gameplay experience by introducing visually captivating and interactive elements. The project uses the 1995 board game Settlers of Catan as a building block and basic formula for one example of how AR can be used to bridge the engagement gap (Catan 1995). The project involves crafting interactive board game elements that leverage AR technology to enhance gameplay, creating a visually captivating and dynamic experience for players. As previously touched on, ‘The Blend’ as it is referred to, is the concept of finding a balancing point between integration and redesigning the older game elements and pieces. The goal is to create elements of physical tools that feel familiar to older audiences who recognize and enjoy the original game, while also providing a new spin on those elements that will be able to reach and draw in a demographic of new players who are familiar with a more modern form of entertainment.

Freezing of gait is a common, treatment-resistant symptom of Parkinson Disease that cancause falls and reduce quality of life. Previous research has demonstrated benefits ofcueing on freezing of gait. However, no effective cue modality has been easily adaptableto daily, real-world use. Visuospatial augmented reality cues are promising in this regard,but have not shown reduction in freezing of gait, possibly due to cue design andlimitations of past augmented reality headsets. The purpose of this study was to developand test a versatile new augmented reality cue on a recently released, field-leadingaugmented reality headset. Also developed were several novel visuospatial cue activationstrategies which were compared to established modalities.The cue was tested in a population with Parkinson Disease and freezing of gait.While no single strategy significantly reduced freezing of gait across the cohort, eachparticipant had on average at least two cueing strategies which significantly reduced timespent freezing compared to a physical cue and control (no cue). Those strategies alsoimproved mean stride length, a marker of gait health. Survey results indicate that themajority of the cohort found the cue to be effective and, with an improved headset, woulduse the cue in their daily life.

Shopping is important in our life, especially food shopping. Almost every family needs to go shopping once per week and spend lots of time and energy on it. This project proposes the design of an application to help people have a better shopping experience by saving time, saving energy, and saving money. The application combines online shopping and in-person shopping in a new shopping mode. People don’t need to pay the expensive delivery fee and wait a long time to get food which could be unfresh. And they also don’t need to spend lots of time at the supermarket, which could cost lots of time by walking around and finding items. This application can solve all shopping problems people might face today. My design intends to help people have a better shopping experience and save precise time on weekends.

General Physics laboratory experiments can be time-consuming to set up for each class.Additionally, students often have a specific room and limited time to conduct the experiment and draw conclusions. One advantage of moving the labs to an augmented reality setting is that setup time can be lowered. The main motivation, however, would be to allow students more time to explore the experiments on their own outside of class. Because the laboratory experiment is a computer simulation, it could provide students with options not available in a physical lab, for example, one could alter the acceleration due to gravity. With added capabilities, augmented reality laboratory experiments could be used either as a replacement for, or an extension of, traditional laboratory experiments. This paper explores the feasibility of using augmented reality to replace Physics laboratory experiments. A one-dimensional motion Physics laboratory experiment was recreated in the Unity game engine and deployed to the Microsoft Hololens 2. Study participants worked through instructions which had them collect data and perform calculations based on that data. The experiment was conducted as an exploratory study to get feedback on the user interface and how well participants were able to complete the tasks, and to gauge the general response to using augmented reality for a Physics laboratory experiment. Based on early findings, an augmented Physics laboratory experiment appears to provide a suitable replacement, but the provided material and available toolset contribute to the effectiveness of the experience.

Analysts often work with large complex non-quantitative datasets in order to better understand concepts, themes, and other forms of insight contained within them. As defined by Pirolli and Card, this act of sensemaking is cognitively difficult, and is performed iteratively and repetitively through various stages of understanding. Immersive analytics has purported to assist with this process through putting users in virtual environments that allows them to sift through and explore data in three-dimensional interactive settings. Most previous research, however, has focused on quantitative data, where users are interacting with mostly numerical representations of data. We designed Immersive Space to Think, an immersive analytics approach to assist users perform the act of sensemaking with non-quantitative datasets, affording analysts the ability to manipulate data artifacts, annotate them, search through them, and present their findings. We performed several studies to understand and refine our approach and how it affects users sensemaking strategies. An exploratory virtual reality study found that users place documents in 2.5-dimensional structures, where we saw semicircular, environmental, and planar layouts. The environmental layout, in particular, used features of the environment as scaffolding for users' sensemaking process. In a study comparing levels of mixed reality as defined by Milgram-Kishino's Reality-Virtuality Continuum, we found that an augmented virtuality solution best fits users' preferences while still supporting external tools. Lastly, we explored how users deal with varying amounts of space and three-dimensional user interaction techniques in a comparative study comparing small virtual monitors, large virtual monitors, and a seated-version implementation of Immersive Space to Think. Our participants found IST best supported the task of sensemaking, with evidence that users leveraged spatial memory and utilized depth to denote additional meaning in the immersive condition. Overall, Immersive Space to Think affords an effective sensemaking three-dimensional space using 3D user interaction techniques that can leverage embodied cognition and spatial memory which aids the users understanding.

Immersive technologies are widely used in interactive media art and storytelling today. A high-quality virtual environment and avatar can provide the audience with a strong sense of presence and ownership of the virtual body, evoking empathetic outcomes from the experience. However, the majority of immersive art we encounter today shows a strong favoring of visual and auditory sensation over interactions. They provide the audience with a limited number of ways to participate in an immersive narrative, leaving them a sense of being an “outsider” that breaks the connection between the audience’s virtual avatar and the virtual environment. Starting with Nicolas Bourriaud’s relational aesthetics for art creation and taking human relations and their social context as the main conceptual and practical point of departure for experience design, I develop artistic and technical strategies for creating immersive experiences that place the audience at the center to let them activate the narrative content and cultural meanings. My approach is to provide the user novel ways to relate themselves to what they see on a screen, and to allow for direct participation and engagement. Through an increased level of interactivity, co-presence and plausibility in immersive environments, my work aims to help the audience develop new relationships with other people, with virtual entities, and with AI agents. The hybrid systems introduced in this thesis describe novel ways of creating relational experiences by techniques learned from different domains such as soft robotics, quantum computing and social science. I also demonstrate authoring interface design that can be more widely adopted by the creative community to overcome the challenges often faced in immersive and interactive narrative compositions.

In recent years, Augmented Reality (AR) glasses have emerged as groundbreaking wearable devices, enabling a harmonious integration of virtual information with the real world. However, challenges remain in enhancing the optical systems that underpin AR glasses, including resolution and field of view limitations. This research focuses on leveraging metasurfaces, composed of subwavelength nanostructures, to design and optimize advanced optical components for AR glasses. By tailoring the properties of metasurfaces, such as amplitude, phase, and polarization control, we aim to overcome limitations and enhance the resolution, field of view, and overall visual performance of AR glasses. To achieve truly immersive AR experiences, improving the optical performance is essential. This involves minimizing aberrations, enhancing contrast, reducing glare, and optimizing light transmission efficiency. Metasurfaces offer unique capabilities for precise manipulation of light at the nanoscale, enabling the creation of highly efficient and compact optical components. By integrating metasurface-based elements, such as gratings, lenses, beam splitters, and polarization controllers, we can enhance resolution, expand the field of view, and improve overall optical performance. Furthermore, metasurfaces offer multifunctionality and the potential for dynamic control. By combining multiple functionalities within a single metasurface-based element, the complexity of optical systems can be reduced, leading to more compact and lightweight AR glasses. Additionally, incorporating tunable or switchable elements within metasurfaces enables real-time control of optical properties, allowing for adaptation to changing environmental conditions and advanced functionalities like adaptive focus and depth perception. This research also explores the challenges of small eyebox in Maxwellian view systems, which restricts user head movement and comfort. Innovative methods utilizing input angle modulation at the metasurface optical element (MOE) are proposed to increase the eyebox size while maintaining optical performance. The study includes the design and optimization of metasurface gratings, algorithm development for optimizing optical elements, implementation of a high-resolution full-color prototype, and investigation of imaging techniques like ultrafast light field tomography (LIFT). By combining these research efforts, this study aims to significantly advance the design and optimization of metasurface-based diffractive waveguide systems for AR applications. The outcomes have the potential to revolutionize the field of augmented reality, pushing the boundaries of optical resolution, addressing limitations, and enhancing the overall user experience. The findings contribute to the advancement of AR glasses, enabling new possibilities in industries such as education, entertainment, healthcare, and beyond.

With the increasing adoption of Augmented Reality/Virtual Reality (AR/VR - XR) systems, security, privacy, and quality concerns attract attention from both academia and industry.We identify four key problems in XR applications and make contributions respectively.For XR Quality of experience: AR applications suffer from spatial inconsistency where virtual objects "drift or "jump" in space. We develop an automated method to evaluate the spatial inconsistency of AR applications. Web-based XR suffers from long latency in downloading virtual content to be displayed on a web browser. We develop a technique to divide up and prioritize virtual objects to be downloaded in a manor to minimize the time to first correct image. For XR Security: Head mounted displays track the user's head pose at all times in order to render content correctly. We show that these head poses can be used to help infer user hand-typed words in order to steal private information. Multi-use AR applications construct a "shared-state" using image sequences and GPS data. We show that these shared-states can be poisoned allowing for the reading and writing of information to and from arbitrary locations by a malicious user.

Augmented Reality (AR) displays show virtual information spatially in the physical world. For example, in the surgical domain, AR displays can overlay medical imaging on top of the patient to help surgeons distinguish healthy tissue from cancerous tissue. This affordance of adding visual information near the area of action has led to a gamut of AR applications with one caveat: their design assumes a flawless alignment between the virtual information and its physical context. If this alignment fails, the system might mislead users by presenting the virtual information out of context.Fortunately, technical advancements in tracking, calibration, and optics have helped improve and create a more seamless augmented reality experience; however, even in these improved systems, alignment failure will lead to user-task performance errors. In my dissertation, I argue that AR interface design needs to account for such misalignments and limitations of AR technology. First, I investigated the case of a situated checklist for cardiopulmonary resuscitation (HoloCPR), which helped deliver accurate guidance even in the presence of tracking errors. Then, I investigated a telementoring system for trauma procedures (ARTEMIS) which helped remote surgeons communicate with novices despite a constant registration error. Finally, I showed that slight optical deviation can drastically impact overall user-task performance and cognitive workload, even in an idealized environment with perfect tracking and calibration.I sum the findings of this final piece, along with the systems designed and developed in this dissertation, to point in the direction of a new type of AR interface building block: contextual aids and how they may have significant implications for AR interface design for surgical procedures, paving the way for the development and integration of more advanced AR systems in surgical practice.

This thesis proposes a novel interactive immersive analytics tool and methods to interrogate the cancer patient cohort in an immersive virtual environment, namely Virtual Reality to Observe Oncology data Models (VROOM). The overall objective is to develop an immersive analytics platform, which includes a data analytics pipeline from raw gene expression data to immersive visualisation on virtual and augmented reality platforms utilising a game engine. Unity3D has been used to implement the visualisation. Work in this thesis could provide oncologists and clinicians with an interactive visualisation and visual analytics platform that helps them to drive their analysis in treatment efficacy and achieve the goal of evidence-based personalised medicine. The thesis integrates the latest discovery and development in cancer patients’ prognoses, immersive technologies, machine learning, decision support system and interactive visualisation to form an immersive analytics platform of complex genomic data. For this thesis, the experimental paradigm that will be followed is in understanding transcriptomics in cancer samples. This thesis specifically investigates gene expression data to determine the biological similarity revealed by the patient's tumour samples' transcriptomic profiles revealing the active genes in different patients. In summary, the thesis contributes to i) a novel immersive analytics platform for patient cohort data interrogation in similarity space where the similarity space is based on the patient's biological and genomic similarity; ii) an effective immersive environment optimisation design based on the usability study of exocentric and egocentric visualisation, audio and sound design optimisation; iii) an integration of trusted and familiar 2D biomedical visual analytics methods into the immersive environment; iv) novel use of the game theory as the decision-making system engine to help the analytics process, and application of the optimal transport theory in missing data imputation to ensure the preservation of data distribution; and v) case studies to showcase the real-world application of the visualisation and its effectiveness.

This study explores whether augmented reality (AR) is an effective form of visual communication through poster design. Effectiveness is measured here by how engaging viewers find posters to be and how well they are able to recall information presented on the posters.Secondary research was conducted to understand what affordances AR provides that may improve engagement and information recall. User interviews and card sorts established a framework for how viewers think about augmented reality techniques—namely whether or not the augmentation breaks the original frame of reference and how relevant the augmentation is to the poster’s message.Using this framework, four tests were conducted using experimental groups, which were exposed to augmented reality communications, and control groups, which received traditional print communications. These tests included user observations, interviews, and surveys to measure the viewers’ perceived level of engagement with the communications as well as their ability to recall information immediately following the viewings and one week later.The study finds that the use of augmented reality in poster communication does significantly increase a viewer’s perceived level of engagement with the communication, but also significantly decreases their ability to recall information from the poster. The use of augmented reality distracts the viewer and interrupts their ability to encode information at a semantic level, which is a process that improves information recall and is needed to create meaning. The study also finds that the bio-cost of viewing AR posters is currently large enough that viewers are unlikely to engage with them by choice.

Nowadays, people prefer browsing images and videos on their phones rather than reading a book (Thain 2018). Even if they decide to read something, they are still heavily distracted by electronic devices. In my thesis, I’m trying to balance the visual and text information and help people to stay more focused while reading. My solution: Focus is an AR glasses system that recognizes the contents users are viewing and records users’ reading and image browsing time. And users have to earn image/video time by reading the text. The glasses block images/videos if they run out of image time.

The demand for technology integration has presented teachers with new pedagogical choices for designing engaging learning experiences for students. These technology choices are becoming abundant and varied, confronting teachers and administrators with determining how effective these methods are for teaching. This research was focused on one of these choices, whether including mixed reality (MR) technologies enhanced the learning experience for college students in a field-based course. The experiential learning theory (Kolb, 1984) provided the framework for understanding the learning process through the experiential learning cycle; a cyclical process of starting with a concrete experience, reflecting on that experience, formulating new concepts, then applying what was learned. This theoretical framework was used to explore the research questions of how MR technology might enhance the learning process, enrich the reflections process, and determine if the students perceived these MR enhancements as useful. This constructivist process for learning was the basis for developing the MR field guide which provided the tool for examining the student experience while they participated in a soil science field study course. This research followed a constructivist-interpretive paradigm using a qualitative, exploratory case study methodology Merriam & Tisdell (2015). Interviews and surveys were used to illuminate the experience the students were having while using the MR field guide. Key findings indicated that the inclusion of the MR field guide has the potential to improve the learning experience and the reflective observation needed to influence learning. The research found that the MR technology facilitated by the MR field guide had both advantages and challenges in enhancing the learning experience. Student participants felt MR technology provided enhanced learning opportunities if certain challenges were overcome. The insights gained from the findings of this study will add to the knowledge for developing more effective teaching strategies when using MR technology and provide guidance to future efforts for technology integration in environmental education.

In the process of linguistic landscaping, spaces are transformed into textual and semiotic representations linked to particular social uses, cultural meanings, and historic narratives. Recently, digital technology has been used to create additional layers of semiotic representation in linguistic landscapes. This dissertation investigates multiple layers of digital and physical representation at six United States’ park spaces, with a particular focus on heritage tourism sites, and analyzes the social meaning and narratives of tourism constructed both by individual layers and within their relationships. Photo-based methods are used to collect data, both by documenting the representations of sites as filtered through the AR platform, Niantic’s Pokémon GO, as well as through photographing the physical sites in-person. This process of ground truthing revealed the erasure of Indigenous histories, as well as amplification of eurocentric histories, which was perpetuated by both the physical and digital layer. The results also revealed a simplification of representations via the digital layer, as well as the role of corporate mediation in regulating space representation. These results are significant in revealing the role of augmented reality in portraying a particular sanitized and gamified version of the spaces. This project also suggests a need within linguistic landscaping research for more engagement with the interdisciplinarity of landscape studies as a whole, particularly, in a US parks context, for gaining a more precise and holistic view of the complex histories and contexts that are obscured by both physical and digital representations.

The internet and mobile devices allow us to consume digital data at every corner. We also encounter data in the real world between these interactions with the virtual world. When we see movie posters, pass street signs, or fill out a medical form, we observe data in the real world instead of the virtual world. The rise of augmented reality (AR) technologies allows us to consider how we can improve interactions with this real-world data. For example, consider a restaurant-goer who wishes to find which items are popular on the menu they are looking over. Currently, the person must look up the ratings online themselves. In this thesis, we propose methods that enable the person to view the menu with their mobile device and query for the menu item ratings by interacting directly with the real-world view of the menu.The work in this thesis enables systems that capture real-world data through an AR device, enhance the real-world data with data from remote sources, and allow a user to query real-world data in augmented reality interactively. The first part of this thesis describes a visual querying grammar and the implementation of a system for querying tabular data in the real world. The second part of this thesis showcases a declarative framework for developing applications that join real-world data with data from remote stores.

Emerging mobile systems such as robots or Augmented Reality glasses increasingly interact with their physical surroundings. This means they require efficient computing platforms to continuously process a high volume of sensory information in a limited-energy battery-powered device. Highly specialized Domain-Specific System on Chips (DSSoCs) have been recently proposed as a solution to provide this efficiency. However, although efficient, the complexity of DSSoCs driven by a high count of hardware intellectual property (IP) blocks results in a long development time, concretely a long design exploration and implementation time. This is because a high IP count enlarges the design space, makes the search for optimal design difficult, and thus lengthens the exploration time. In addition, each new IP in the system increases the implementation time, i.e., the time associated with converting an algorithm to a hardware IP, verification, and integration. These two problems only worsen as the future system’s intelligence and, thus, computational need expands. This dissertation demonstrates how to mitigate the DSSoC’s long exploration and implementation time by employing efficient resource selection and resource management techniques. We demonstrate how to lower the exploration time using an agile design space exploration (DSE) framework that efficiently selects system resources. Similar to other DSEs, our framework consists of a simulator and an exploration heuristic. In contrast to the state-of-the-art simulators that are either too slow or inaccurate and thus infeasible for sufficient coverage of DSSoCs’ large design space, our simulator is highly suited for DSSoCs as it combines the agility of analytical models and the accuracy of the phase-driven models. Our result shows that this methodology speeds up the state-of-the-art transactional models by 8400x while only incurring a small 1.5% error for a host of complex SoCs. Our DSE also lowers the exploration time by deploying an agile search heuristic that efficiently navigates DSSoCs’ large design space. In this dissertation, we highlight two features for an efficient search heuristic, namely 1) joint-optimization capability to exploit cross-boundary optimizations and 2) architectural awareness to prevent blind traversal of the design space. State-of-the-art solutions either lack the former or the latter. In contrast, our methodology combines the two and speeds up the convergences of the baseline, i.e., the classic simulated annealing (SA) and more modern Multi-Objective optimistic Search (MOOS) by 62× and 35×, respectively. It also improves their Quality Gain and Pareto Hyper Volume. We also demonstrate how to lower the implementation time by replacing high-effort IP customization solutions with low-effort environment-aware resource management ones. Our solution uses the physical environment’s heterogeneity, a prominent characteristic of the mobile edge domain, to its advantage and dynamically manages system resources to improve efficiency. To this end, first, we show that various spatial heterogeneity factors, such as environment congestion, impact the processing payload. Then, we establish that designs that ignore this heterogeneity incur system-level performance and energy degradation and thus require specialized IPs to solve their efficiency problem; However, introducing IPs increases the implementation time and effort. Thus, to address this problem, we provide a resource management runtime that dynamically exploits said compute-environment synergy and improves the system efficiency without introducing customized IPs. We implement our runtime in Robot Operating System (ROS) middleware and evaluate it for autonomous drones without the loss of generality. We compare our runtime with a spatially oblivious system, typical of traditional commercial deployments, whose parameters are statically set at design time. We show that exploiting spatial heterogeneity leads to 4.5x improvement in mission time, 4x improvement in energy efficiency, and 36% reduction in CPU utilization. The contributions made in this dissertation will likely have a long-term impact as, with the end of Moore’s law, highly specialized DSSoCs have shown promise to address general systems’ lack of efficiency. Thus, understanding and further providing techniques to mitigate DSSoCs’ design and development issues are of high importance.

Orientation Device is a tool for understanding the other towards recognizing alternative possibilities, for care and compassion, for expanding our culturally and politically bounded mindset, a tool of vital nausea and questioning compulsory heterosexuality. The work is a series of augmented reality (AR) experiences for mobile device that allow the audience to participate in documentary queer performances in any private or public setting. These immersive experiences challenge our perception of space. The LGBTQIA+ community is often disoriented within heteronormative spaces, and this work reverses that dichotomy by disorienting the audience. As a cisgender creator, I invite queer performers, artists, poets, and thinkers who express their identity in their creative practices. As the AR medium is widely distributable, I want to give the participants the potential opportunity to present themselves to a wide international audience through the poetics of augmented reality and documentary video holograms.

The purpose of this causal-comparative study was used to analyze the hypothesized differences among inclusion teachers’ sense of self efficacy (TSES) based on their highest level of degree completion (bachelor’s, master's, master's plus). In an era of educational reform, students with specific learning disabilities (SLD) continue to lag behind regular education counterparts in all schooling environments. Schools in the southwestern region of Tennessee are now servicing a growing number of SLD-diagnosed students with a paucity of research addressing teacher's impact on SLD literacy scores. Participants in the study comprised 59 English language arts (ELA) inclusion teachers in 12 public middle school settings. To address this gap in the research, data were examined using a one-way analysis of variance (ANOVA). Survey Monkey and TSES long form was used to collect demographic and professional information. Results of the one-way ANOVA revealed no statistically significant differences among inclusions teachers with different levels of education. Future recommendations for research in virtual reality (VR) and augmented reality (AR) can enhance instructional practices while building teacher's confidence to motivate students' interest in ELA achievement in all public-school learning environments.

Ruminations is an experimental exhibition which seeks to explore the integration of technology into traditional art. By combining traditional woodblock prints with augmented reality, Ruminations attempts to evoke empathy from the viewer regarding the topic of mental illness. Each AR experience illustrates, though imagery and animation, a different mental illness or psychological phenomenon. The viewer sees each experience separately over a printed background, expanding on the traditional frame and reaching into the world around us.