Search

This portfolio of compositions uses novel biometrics from gestural interfaces called Myo armbands to compose five pieces of electroacoustic music within a continuum of spatial environments, ranging from the real to the virtual. The portfolio achieves this by processing biometric data from the Myo armbands within developed software (using Max 8), applying machine learning to such data (via Wekinator) and mapping predictive outputs to audiovisual materials across the spatial continuum. The results of this approach show how unique compositional affordances can be created when using biometrics within different spatial environments and how implementing machine learning methods can help to create such opportunities. The outcomes are presented via five composed portfolio works, three published papers within the scientific community, and numerous music performances navigating the overlapping areas of music composition and computer science. By using novel biometrics to compose music within a continuum of space, this interdisciplinary research topic contributes original knowledge to both the fields of music composition and computer science (human-computer interaction).

Over the last decade, modern transportation systems have evolved to a considerably larger scope and higher complexity, brought by a wave of innovation in technologies and a revolutionary shift in how mobility is perceived and realized – giving rise to a strong orientation towards services. This thesis imagines a time when cars can drive fully autonomous. Drivers no longer need to drive. So, what else can we offer passengers during the time when they are sitting in the car? The purpose of this project is to introduce a concept of mainly using Augmented Reality and AI recognition to create a virtual tour guide in autonomous cars that will provide educational knowledge and immediate assistance through AR windshields and side windows. Imagine a time when you are sitting in an autonomous car, and you get to know your surroundings through AR windshield and side windows. People can now explore the world as if a real tour guide is always around.

In almost every field, there is a need for strong interpersonal skills. This is especially true in fields such as medicine, psychology, and education. For instance, healthcare providers need to show understanding and compassion for LGBTQ+ and BIPOC (Black, Indigenous, and People of Color), or individuals with unique developmental or mental health needs. Improving interpersonal skills often requires first-person experience with expert evaluation and guidance to achieve proficiency. However, due to limited availability of assessment capabilities, professional standardized patients and instructional experts, students and professionals currently have inadequate opportunities for expert-guided training sessions. Therefore, this research aims to demonstrate leveraging technological advances in mobile computing (MC), automatic speech recognition (ASR), natural language processing (NLP) and augmented reality (AR) technology to address such limitations. Our result is a mobile application created in collaboration with a psychology expert to provide an innovative training/learning serious game.

Industry 4.0 technology (I4.0) is inescapable. It transforms the way businesses and customers interact and revolutionizes how organizations produce goods and services (SAP Insights, 2020). It requires a level of agility that many organizations do not possess. Defending against disruptive business models is no longer enough. Organizations must be nimble to optimize assets and resources in response to adversity. In March 2020, the coronavirus (COVID-19) pandemic ushered a devastating blow to the U.S. economy and job market with pervasive shocks that continue to be a business threat. In response, many organizations are accelerating automation, digitization, and communication capabilities to close the gap and connect with customers. This dissertation examined the cross-industry adoption of the nine most common Industry 4.0 technologies: big data, artificial intelligence, cloud computing, the internet of things, cybersecurity, 3-D printing, autonomous technology, augmented reality, and blockchain. This descriptive study explored factors of I4.0 adoption across industries and organizational sizes during a national pandemic. The study sought to reveal “what” factors contributed to the adoption of Industry 4.0, “what” industry patterns exist, “what” effect COVID-19 had on these concepts. A quantitative method was used to examine the relationship between factors. An online survey was administered to a Qualtrics panel of 520 business owners and executives to capture perceptions, knowledge, and insights. A binary logistic regression analysis was performed. The results of this study inform a cross-industry framework of I4.0 technology adoption, which includes contributing factors. The findings also showed COVID-19 was less an accelerant of adoption but rather, the industry sector was a greater influencer.

This dissertation rethinks the problem of robot imitative learning given human demonstrations and proposes a holistic framework to unify three challenges: (i) understanding the actions being imitated, (ii) producing proper imitative behaviors, and (iii) interacting effectively with humans. Considering a complex manipulation task of opening medicine bottles, some key actions of pushing or squeezing are critical in unlocking the bottles, but can hardly be recognized from pure visual observations. Therefore, a glove-based system is firstly presented together with a demonstration collection pipeline to understand the actions from a functional perspective that incorporates hand movements and goal states and from a physical perspective that focuses on the forces to reach the states. This heterogeneous information is integrated by a Temporal And-Or Graph (T-AOG) grammar representation, which also captures the hierarchical structure of the task. Sampling from the T-AOG generates a valid action sequence to accomplish the task. To transfer this skill to a robot, a mirroring approach is then proposed for a robot to infer functionally equivalent actions that can produce a similar force pattern in a physics-based simulation and achieve the same goal in changing object states, naturally bridging the action perception and production in robot imitation. In addition, using such a grammar representation is advantageous in tracking object states and accumulating robot knowledge from multiple views over a long period of time. Based on these, a joint inference algorithm is proposed to infer human (false-)beliefs and overcome the ambiguity in visual detections. Finally, this dissertation studies how different forms of explanation generated from the representation prompt human trusts in a robotic system and develops an Augmented Reality (AR) interface that allows users to interactively supervise a robot’s decision-making process and intervene by patching its knowledge represented by a T-AOG. Having a T-AOG representation as the core, this dissertation seeks to unify the perception, learning, planning, and interaction problems in robot imitation.

Common Objects in Context for bridge inspection (COCO-Bridge) was introduced for use by unmanned aircraft systems (UAS) to assist in GPS denied environments, flight-planning, and detail identification and contextualization, but has far-reaching applications such as augmented reality (AR) and other artificial intelligence (AI) platforms. COCO-Bridge is an annotated dataset which can be trained using a convolutional neural network (CNN) to identify specific structural details. Many annotated datasets have been developed to detect regions of interest in images for a wide variety of applications and industries. While some annotated datasets of structural defects (primarily cracks) have been developed, most efforts are individualized and focus on a small niche of the industry. This effort initiated a benchmark dataset with a focus on structural details. This research investigated the required parameters for detail identification and evaluated performance enhancements on the annotation process. The image dataset consisted of four structural details which are commonly reviewed and rated during bridge inspections: bearings, cover plate terminations, gusset plate connections, and out of plane stiffeners. This initial version of COCO-Bridge includes a total of 774 images; 10% for evaluation and 90% for training. Several models were used with the dataset to evaluate model overfitting and performance enhancements from augmentation and number of iteration steps. Methods to economize the predictive capabilities of the model without the addition of unique data were investigated to reduce the required number of training images. Results from model tests indicated the following: additional images, mirrored along the vertical-axis, provided precision and accuracy enhancements; increasing computational step iterations improved predictive precision and accuracy, and the optimal confidence threshold for operation was 25%. Annotation recommendations and improvements were also discovered and documented as a result of the research.

Inspired by continuing developments in artificial intelligence, this creative thesis comprises an overview of conversational artificial intelligence and an exploration of how education and entertainment overlap in the broad genre of science plays. I argue that the combination of education and entertainment can be mutually beneficial, and that this relationship should be exploited to increase public accessibility to information. Moreover, this thesis documents the process of writing, producing, directing, and designing a new work entitled Time to Think: A Short Play about Ignorance & Bliss. The original script, including marginal director’s notes, is appended.

Human-Centered Robotics (HCR) is a research area that focuses on how robots can empower people to live safer, simpler, and more independent lives. In this dissertation, I present a combination of two technologies to deliver human-centric solutions to an important population. The first nascent area that I investigate is the creation of an Intelligent Robot Instructor (IRI) as a learning and instruction tool for human pupils. The second technology is the use of augmented reality (AR) to create an Augmented Reality Instruction (ARI) system to provide instruction via a wearable interface. To function in an intelligent and context-aware manner, both systems require the ability to reason about their perception of the environment and make appropriate decisions. In this work, I construct a novel formulation of several education methodologies, particularly those known as response prompting, as part of a cognitive framework to create a system for intelligent instruction, and compare these methodologies in the context of intelligent decision making using both technologies. The IRI system is demonstrated through experiments with a humanoid robot that uses object recognition and localization for perception and interacts with students through speech, gestures, and object interaction. The ARI system uses augmented reality, computer vision, and machine learning methods to create an intelligent, contextually aware instructional system. By using AR to teach prerequisite skills that lend themselves well to visual, augmented reality instruction prior to a robot instructor teaching skills that lend themselves to embodied interaction, I am able to demonstrate the potential of each system independently as well as in combination to facilitate students' learning. I identify people with intellectual and developmental disabilities (I/DD) as a particularly significant use case and show that IRI and ARI systems can help fulfill the compelling need to develop tools and strategies for people with I/DD. I present results that demonstrate both systems can be used independently by students with I/DD to quickly and easily acquire the skills required for performance of relevant vocational tasks. This is the first successful real-world application of response-prompting for decision making in a robotic and augmented reality intelligent instruction system.