Search

This portfolio of compositions uses novel biometrics from gestural interfaces called Myo armbands to compose five pieces of electroacoustic music within a continuum of spatial environments, ranging from the real to the virtual. The portfolio achieves this by processing biometric data from the Myo armbands within developed software (using Max 8), applying machine learning to such data (via Wekinator) and mapping predictive outputs to audiovisual materials across the spatial continuum. The results of this approach show how unique compositional affordances can be created when using biometrics within different spatial environments and how implementing machine learning methods can help to create such opportunities. The outcomes are presented via five composed portfolio works, three published papers within the scientific community, and numerous music performances navigating the overlapping areas of music composition and computer science. By using novel biometrics to compose music within a continuum of space, this interdisciplinary research topic contributes original knowledge to both the fields of music composition and computer science (human-computer interaction).

Dementia affects more than 850,000 people in the UK, and this figure continues to rise year on year. People living with dementia often have difficulties completing activities of daily living (ADLs), leading to a reliance on family or professional carers. However, assistive technology, such as task prompting tools, can support people living with dementia to maintain their independence and live at home for longer. Augmented Reality (AR) is an increasingly prevalent technology and has been used as a task prompting tool in industrial settings to support complex maintenance and assembly tasks. The use of AR for task assistance has also been identified as a promising area for development in Domestic AR. Despite the use of AR as a task prompting tool in industrial settings and the potential of AR in domestic settings, relatively little is known about the efficacy of augmentations for ADLs. The work in this thesis aims to provide an initial exploration into the use of AR as a task prompting tool to support people living with dementia to complete ADLs at home. Multiple stakeholders, including health professionals, a general adult population, older adults without cognitive impairment, and people living with dementia and their family carers, have been included to develop a holistic understanding of the use of AR in a domestic setting, based on four studies carried out for this project. The first study consisted of in-person interviews with professionals with experience of working with people living with dementia. The second study was a lab experiment with older adults to compare four AR visual prompting techniques to prompt five basic actions found in many ADLs. The third study involved the co-design of AR prompts in a kitchen context, and their evaluation with a general adult population using an online survey. The final study consisted of online interviews with people living with dementia and their family carers to explore the results of the previous three studies and how AR could be beneficial from the point of view of people with lived experience of dementia. The overall findings show that AR as a tool to support task prompting of domestic tasks was received positively by the participants of these studies. A combination of text, audio, and a ghosthand image demonstrating the action to carry out could be most beneficial for people with dementia, but AR prompts should be easily customisable to cater for different abilities, preferences, and personalities. Furthermore, early introduction of AR will be key for uptake when the technology has been developed further. The potential of domestic AR to improve the lives of people affected by dementia and those that support them with ADLs is considered as motivation for future work in this promising research area.

Augmented reality (AR) is the enhancement of our perception of the world around us. It promises the ability to simultaneously interact with a virtual environment and real objects, providing an intuitive method to interact with data. However, AR displays must be lightweight and compact, and be able to display bright, high-resolution 3D images. Current AR displays are limited by bulky-form factors with a limited field of view (FOV), and the inability to display high quality 3D images without introducing side effects such as nausea and eyestrain. In the first part of this thesis an afocal Maxwellian display is analysed and modelled, the small eyebox limitation examined and a solution proposed. A prototype head mounted display unit was constructed as a demonstration. The display performance was analysed objectively and subjectively, and found to perform well over a wide accommodation range. However, the system is mechanically complicated to provide for a large IPD demographic, and the implementation of the eyebox expansion is too limited. An alternative curved reflector system was proposed to reduce weight, increase FOV, and increase the eyebox. The final part of this research examines waveguides as an AR display technology. The limitations of current systems are evaluated, and a novel switchable waveguide architecture proposed where light can be selectively directed towards the eye to form an image time sequentially. This enables higher quality images, with a wider FOV and increased efficiency, from a more transparent waveguide. A liquid crystal diffraction grating is developed as a switching element with a simulated diffraction efficiency of 20% and a geometry optimised for easy industrial manufacture. The possibility for such a system to include an integrated lens function is also considered, and an 80° FOV design presented.

The creation of an effective construction schedule is fundamental to the successful completion of a construction project. Effectively communicating the temporal and spatial details of this schedule are vital, however current planning approaches often lead to multiple or misinterpretations of the schedule throughout the planning team. Four Dimensional Computer Aided Design (4D CAD) has emerged over the last twenty years as an effective tool during construction project planning. In recent years Building Information Modelling (BIM) has emerged as a valuable approach to construction informatics throughout the whole lifecycle of a building. Additionally, emerging trends in location-aware and wearable computing provide a future potential for untethered, contextual visualisation and data delivery away from the office. The purpose of this study was to develop a novel computer-based approach, to facilitate on-site 4D construction planning through interaction with a 3D construction model and corresponding building information data in outdoor Augmented Reality (AR). Based on a wide ranging literature review, a conceptual framework was put forward to represent software development requirements to support the sequencing of construction tasks in AR. Based on this framework, an approach was developed that represented the main processes required to plan a construction sequence using an onsite model based 4D methodology. Using this proposed approach, a prototype software tool was developed, 4DAR. The implemented tool facilitated the mapping of elements within an interactive 3D model with corresponding BIM data objects to provide an interface for two way communication with the underlying Industry Foundation Class (IFC) data model. Positioning data from RTK-GPS and an electronic compass enabled the geo-located 3D model to be registered in world coordinates and visualised using a head mounted display fitted with a ii forward facing video camera. The scheduling of construction tasks was achieved using a novel interactive technique that negated the need for a previous construction schedule to be input into the system. The resulting 4D simulation can be viewed at any time during the scheduling process, facilitating an iterative approach to project planning to be adopted. Furthermore, employing the IFC file as a central read/write repository for schedule data reduces the amount of disparate documentation and centralises the storage of schedule information, while improving communication and facilitating collaborative working practices within a project planning team. Post graduate students and construction professionals evaluated the implemented prototype tool to test its usefulness for construction planning requirements. It emerged from the evaluation sessions that the implemented tool had achieved the essential requirements highlighted in the conceptual framework and proposed approach. Furthermore, the evaluators expressed that the implemented software and proposed novel approach to construction planning had potential to assist with the planning process for both experienced and inexperienced construction planners. The following contributions to knowledge have been made by this study in the areas of 4D CAD, construction applications of augmented reality and Building Information Modelling; · 4D Construction Planning in Outdoor Augmented Reality (AR) · The development of a novel 4D planning approach through decomposition · The deployment of Industry Foundation Classes (IFC) in AR · Leveraging IFC files for centralised data management within real time planning and visualisation environment.

Freezing of gait is a common, treatment-resistant symptom of Parkinson Disease that cancause falls and reduce quality of life. Previous research has demonstrated benefits ofcueing on freezing of gait. However, no effective cue modality has been easily adaptableto daily, real-world use. Visuospatial augmented reality cues are promising in this regard,but have not shown reduction in freezing of gait, possibly due to cue design andlimitations of past augmented reality headsets. The purpose of this study was to developand test a versatile new augmented reality cue on a recently released, field-leadingaugmented reality headset. Also developed were several novel visuospatial cue activationstrategies which were compared to established modalities.The cue was tested in a population with Parkinson Disease and freezing of gait.While no single strategy significantly reduced freezing of gait across the cohort, eachparticipant had on average at least two cueing strategies which significantly reduced timespent freezing compared to a physical cue and control (no cue). Those strategies alsoimproved mean stride length, a marker of gait health. Survey results indicate that themajority of the cohort found the cue to be effective and, with an improved headset, woulduse the cue in their daily life.

Human errors in healthcare can be fatal. Proper physical assessment of patients to avoidsuch errors is of paramount importance. Incorrect or insufficient assessment of the patientcan cause treatment delays that may lead to negative outcomes. In this dissertation we introduce innovative technology to assist surgeons in patient assessment as well as during thetraining of nurses in order to enhance learning. Technological advancements have madeit possible to visualize overlays of computer-generated 3D models on real-world surfaces.This technology is called augmented reality. Using Steady State Topography (SST) brainimaging to examine the brain activity of people who participated in AR and non-AR tasks,Heather Andrew et. al. [13] demonstrated that visual attention is almost doubled when performing AR tasks when compared to non-AR tasks. Memory retention is also demonstratedto be 70% higher for AR experiences. Other studies show that the long-term memory ofthe learner can be enhanced by using multiple media interactions in the learning process[12]. The ability to project images accurately placed onto real-world physical objects usingheadsets could lead to increased retention of precision performance related to techniquesin the physical assessment of the patient. This particular study deals with using AR to helpsurgeons visualize and assess the internal organs of a patient, particularly the rib cage. Thisstudy also proposes the use of using a similar variation of the application to enable nursingstudents to learn the necessary physical techniques required during the assessment of theheart and the lungs. Students will be able to visualize an overlay of the ribs, heart, andlungs to increase understanding of the accurate placement of devices required for assessingcardiac and respiratory issues using anatomical landmarks.

In almost every field, there is a need for strong interpersonal skills. This is especially true in fields such as medicine, psychology, and education. For instance, healthcare providers need to show understanding and compassion for LGBTQ+ and BIPOC (Black, Indigenous, and People of Color), or individuals with unique developmental or mental health needs. Improving interpersonal skills often requires first-person experience with expert evaluation and guidance to achieve proficiency. However, due to limited availability of assessment capabilities, professional standardized patients and instructional experts, students and professionals currently have inadequate opportunities for expert-guided training sessions. Therefore, this research aims to demonstrate leveraging technological advances in mobile computing (MC), automatic speech recognition (ASR), natural language processing (NLP) and augmented reality (AR) technology to address such limitations. Our result is a mobile application created in collaboration with a psychology expert to provide an innovative training/learning serious game.

With a lineage originating in the days of silent films, visual music, in its current incarnation, is a relatively recent phenomenon when compared to an historically broad field of creative expression. Today it is a time-based audio/visual territory explored and mined by a handful of visual and musical artists. However, an extensive examination of the literature indicates that few of these composers have delved into the associable areas of merging virtual holography and holophony toward visual music composition. It is posited here that such an approach is extremely rich with novel expressive potential and simultaneously with numerous novel challenges. The goal of this study is, through praxis, to instantiate and document an initial exploration into the implementation of holochory toward the creation of visual music compositions. Obviously, engaging holochoric visual music as a means of artistic expression requires an interdisciplinary pipeline. Certainly, this is demonstrated in merging music and visual art into a cohesive form, which is the basis of visual music composition. However, in this study is revealed another form of interdisciplinarity. A major challenge resides with the development of the means to efficiently render the high-resolution stereoscopic images intrinsic to the animation of virtual holograms. Though rendering is a challenge consistent with creating digital animations in general, here the challenge is further exacerbated by the extensive use of multiple reflections and refractions to create complexity from relatively simple geometric objects. This reveals that, with the level of computational technology currently available, the implementation of high-performance computing is the optimal approach. Unifying such diverse areas as music, visual art, and computer science toward a common artistic medium necessitates a methodological approach in which the interdependency between each facet is recognized and engaged. Ultimately, a quadrilateral reciprocative feedback loop, involving the composer's sensibilities in addition to each of the other facets of the compositional process, must be realized in order to facilitate a cohesive methodology leading toward viability. This dissertation provides documentation of methodologies and ideologies undertaken in an initial foray into creating holochoric visual music compositions. Interlaced matrices of contextualization are intended to disseminate the processes involved in deference to composers who will inevitably follow in the wake of this research. Accomplishing such a goal is a quintessential aspect of practice-based research, through which new knowledge is gained during the act of creating. Rather than formulating theoretical perspectives, it is through the praxis of composing holochoric visual music that the constantly arising challenges are recognized, analyzed, and subsequently addressed and resolved in order to ensure progression in the compositional process. Though measuring the success of the resultant compositions is indeed a subjective endeavor, as is the case with all art, the means by which they are achieved is not. The development of such pipelines and processes, and their implementation in practice, are the basic building blocks of further exploration, discovery, and artistic expression. This is the impetus for this document and for my constantly evolving and progressing trajectory as a scholar, artist, composer, and computer scientist.

Traditional forms of experiential marketing have been interrupted by augmented reality. As augmented reality continues to grow, a new ecosystem for brand interactions has revealed a world of branded augmented content or “BX”. BX is a concept design that creates a personalized, immersive connection between brand and user through four AR filters.

Industrial factory floor data has exponentially increased as more and more processes are automated throughout the different industries. Typically, this data is buried inside the machines and only used internally to monitor and correct the process. Industry 4.0 is a new paradigm that requires accessing this data continuously in order to provide monitoring and feedback of the process and integrate this information with other machines or processes on the factory floor. There is also a need for factory personnel to instantaneously acquire machine data from one or more stations using Augmented Reality methods using smart devices. This thesis proposes a novel method for bringing factory floor data into an Industrial IoT environment where it can then be used for further analyses using a Mixed Reality environment in conjunction with smart handheld devices. The method and the process were implemented and validated on workcenters at the Volvo Truck plant in Hagerstown, Maryland. Data from four different stations at the plant was successfully brought into an IIoT platform and then visualized in real-time using an Augmented Reality (AR) experience through a handheld device or an AR headset with the use of spatial anchors. A software solution was created to automate the configuration process based on some basic information about the station and the desired signals. Performance tests were performed and validated to evaluate the efficiency of the data, any latency in the communication, and to test the feasibility of using spatial anchors in an industrial environment.

Motivation and situational interest have proven to be critical factors related to student outcomes. Augmented reality, when leveraged properly, has been demonstrated to be an efficacious instructional vehicle across many academic domains, but little is known about its relationship to motivation and situational interest. Additionally, little empirical research has been performed on augmented reality applications related to the specific domain of history. Merely knowing that a technology, like augmented reality, can produce positive learning outcomes is not enough; understanding how it impacts motivation and situational interest are critical in understanding how and when to leverage this technology.This study analyzed the impact to situational interest and motivation using an augmented reality application that delivers instructional content about the tragic events that occurred on May 4th, 1970 at Kent State University. Using both a qualitative and a quantitative pretest and posttest approach, it was determined that situational interest and motivation were impacted by AR. More specifically, aspects of motivation decreased while situational and content knowledge increased. Through a qualitative approach, this study outlined factors that contributed to these changes. These factors include the feeling of immersiveness and enjoyable multimedia content, as well as negative feelings towards the location finding in the application as well as the applications’ technical reliability

This report examines the ways that users share information using augmented reality (AR). In order to accomplish that goal, the researcher (with the help of their supervisor), conducted interviews of two participants. In those interviews, the participants were asked questions about their sharing behavior on and offline along with their current usage of social media. After the interviews were conducted, the participants were given an AR app called Krikey to use for a period of time. At the end of that time, the participants reconvened with the researcher and answer a few more questions regarding their experience with the AR app and their sharing behavior on the app. The report, through thematic analysis and descriptive statistics, found that the participants were more willing to share information through AR when interacting with groups and individuals with whom they were familiar. On top of that, the users expressed concerns regarding privacy and social interaction within the app. This study provides a preliminary step toward better understanding the sharing habits of users in AR

The value of personal experience in learning is a concept that has been around for thousands of years dating back to the time of Confucius in 450 B.C. Today, personal experience can be accomplished through immersive technology, such as augmented reality, a technology simulating real-world and authentic experiences. Kolb’s Theory of Experiential Learning (1984) and Dale’s Cone of Experience (1946) theorized not only the importance of learning by doing, but that the type or authenticity of the experience is important in learning outcomes, retention, and learner motivation. Immersive technology has advanced to the point that it is not only accessible, but also user friendly. However, research into the impact of immersive technology remains focused in K-12 settings with students as the consumers, rather than creators of authentic experiences. The purpose of this study was to refocus the research to higher education preservice teachers, a unique population who are the potential creators of these experiences. The study investigated if the use of immersive technology in a preservice teacher project-based learning experience influenced knowledge attainment and retention of a key pedagogical concept and if it affected preservice teacher motivation. The key pedagogical concept selected for the study was Universal Design for Learning (UDL). The individual project-based learning experience required preservice teachers to implement these principles into a functional lesson appropriate for their grade level and subject. The study utilized a baseline/post/posttest design and the Instructional Materials Motivational Survey (IMMS) as instruments. The results of the level of knowledge attainment and retention were inconclusive due to underperformance of the baseline/post/posttest instrument. A more functional, hands-on test of the application of the UDL principle would provide more reliable results. In the motivation construct, the results indicated that the type of experience (immersive or interactive) impacted preservice teacher learning. The level of the impact varied based on the levels within the underlying constructs of the motivational measure: attention, relevance, confidence, and satisfaction. The students who utilized the immersive technology showed higher levels of relevance, confidence, and satisfaction. However, the study also revealed a substantial gap in the motivational measure and the need for additional research.

Augmented reality (AR) is a developing technology that is expected to become the next interface between humans and computers. One of the most common designs of AR devices is the optical see-through head- mounted display (HMD). In this design, the virtual content presented on the displays embedded inside the device gets optically superimposed on the real world which results in the virtual content being transparent. Color appearance in see-through designs of AR is a complicated subject, because it depends on many factors including the ambient light, the color appearance of the virtual content and color appearance of the real background. Similar to display technology, it is vital to control the color appearance of content for many applications of AR. In this research, color appearance in the see-through design of augmented reality environment is studied and modeled. Using a bench-top optical mixing apparatus as an AR simulator, objective measurements of mixed colors in AR were performed to study the light behavior in AR environment. Psychophysical color matching experiments were performed to understand color perception in AR. These experiments were performed first for simple 2D stimuli with single color both as background and foreground and later for more visually complex stimuli to better represent real content that is presented in AR. Color perception in AR environment was compared to color perception on a display which showed they are different from each other. The applicability of the CAM16 color appearance model, one of the most comprehensive current color appearance models, in AR environment was evaluated. The results showed that the CAM16 is not accurate in predicting the color appearance in AR environment. In order to model color appearance in AR environment, four approaches were developed using modifications in tristimulus and color appearance spaces, and the best performance was found to be for Approach 2 which was based on predicting the tristimulus values of the mixed content from the background and foreground color.

The development of new data interaction modalities and concurrent advancement in the hardware and database technology has dramatically changed the way users interact with data. Data analysis tasks have now turned into interactive and instantaneous explorations of query-result space. Database systems are not designed to maintain interactive latency while handling the unique and unprecedented workloads posed by the modern interaction modalities. Our work strives to bridge the gap between the end-users and database systems, allowing the users to engage in interactive data exploration without an undue cognitive burden in processing the system feedback. We utilize the user-interaction behavior, the user-interface characteristics, and the human factors that drive the data exploration process to handle the challenges of interactive latency and cognitive overload in different interactive data exploration scenarios. This dissertation presents a middleware component – Flux Capacitor, that insulates the backend from bursty and query-intensive workloads generated by modern web, mobile, touch, and gesture-driven next-generation interfaces. Flux Capacitor uses prefetching and caching strategies informed by the inherent physics-metaphor of UI widgets such as friction and inertia in range sliders and maps, and the typical behavior patterns in user-interaction, enabling low interaction response times while intelligently trading off accuracy when required. We present a data substrate for addressing the unique data management and user interaction challenges posed by Augmented Reality (AR) interfaces for data – DreamStore. The platform incorporates optimizations for AR workload characteristics at various layers of the data stack, treating AR queries as first-class queries. DreamStore provides a user-focus based mechanism to handle visual clutter – a form of cognitive overload and enables interactive latency for user actions through prefetching and caching strategies that utilize the user’s position and movement pattern and the physical placement of AR objects. Even with a performant system supporting interactive latency, a perceived disconnect between the user action and system response can be a source of cognitive overload. For example, in an interactive query session for structured data analysis, it is often difficult to comprehend a succession of transformations, especially for complex queries. Thus, to facilitate understanding of each data transformation and to provide continuous feedback, we introduce Data Tweening. Data Tweening presents to the user a series of incremental visual changes, that call out the user’s attention to the sequence with motion and highlights points of changes with other visual features such as color. The parallel presentation of visual changes encoded in an engaging animation offloads some of the user’s cognitive burden in processing the changes in the result, to their perceptual system. Through empirical evaluation, we demonstrate the efficacy of informing query processing and data presentation for ad-hoc data exploration, with the human factors that drive the interactive data exploration process and the inherent characteristics of the user interface, towards achieving interactive latency and reducing cognitive load

Inspired by continuing developments in artificial intelligence, this creative thesis comprises an overview of conversational artificial intelligence and an exploration of how education and entertainment overlap in the broad genre of science plays. I argue that the combination of education and entertainment can be mutually beneficial, and that this relationship should be exploited to increase public accessibility to information. Moreover, this thesis documents the process of writing, producing, directing, and designing a new work entitled Time to Think: A Short Play about Ignorance & Bliss. The original script, including marginal director’s notes, is appended.

It has been argued that approaches to education should embed learning in activities that reflect the social and physical environments in which the knowledge is relevant. Only recently, did it become possible to situate learning in a variety of novel contexts using augmented reality (AR) games. This study investigates the behaviors of middle school students during their participation in an AR game called Play the Past. The findings of this study show that engagement differed during discrete activities in the game environment and that there was a relationship between the roles that students were assigned and their engagement.

Computer technology helps improve human productivity and performance. We believe that if the computer technology lies, most people are unlikely to notice the deception. We explored whether Augmented Reality can improve a user’s performance, whether an Augmented Reality system will gain human’s trust easily, and whether, when their trust is betrayed, the user is unlikely to notice. To evaluate our three hypotheses, we designed a card sorting experiment involving human subjects with augmented reality (AR) technology. Our experimental results support our hypotheses, with high levels of statistical significance.