Search

The authors present the design and implementation of an exploratory virtual learning environment that assists children with autism (ASD) in learning science, technology, engineering, and mathematics (STEM) skills along with improving social-emotional and communication skills. The primary contribution of this exploratory research is how educational research informs technological advances in triggering a virtual AI companion (AIC) for children in need of social-emotional and communication skills development. The AIC adapts to students’ varying levels of needed support. This project began by using puppetry control (human-in-the-loop) of the AIC, assisting students with ASD in learning basic coding, practicing their social skills with the AIC, and attaining emotional recognition and regulation skills for effective communication and learning. The student is given the challenge to program a robot, Dash™, to move in a square. Based on observed behaviors, the puppeteer controls the virtual agent’s actions to support the student in coding the robot. The virtual agent’s actions that inform the development of the AIC include speech, facial expressions, gestures, respiration, and heart color changes coded to indicate emotional state. The paper provides exploratory findings of the first 2 years of this 5-year scaling-up research study. The outcomes discussed align with a common approach of research design used for students with disabilities, called single case study research. This type of design does not involve random control trial research; instead, the student acts as her or his own control subject. Students with ASD have substantial individual differences in their social skill deficits, behaviors, communications, and learning needs, which vary greatly from the norm and from other individuals identified with this disability. Therefore, findings are reported as changes within subjects instead of across subjects. While these exploratory observations serve as a basis for longer term research on a larger population, this paper focuses less on student learning and more on evolving technology in AIC and supporting students with ASD in STEM environments.

Recognizing the affective state of children with autism spectrum disorder (ASD) in real-world settings poses challenges due to the varying head poses, illumination levels, occlusion and a lack of datasets annotated with emotions in in-the-wild scenarios. Understanding the emotional state of children with ASD is crucial for providing personalized interventions and support. Existing methods often rely on controlled lab environments, limiting their applicability to real-world scenarios. Hence, a framework that enables the recognition of affective states in children with ASD in uncontrolled settings is needed. This paper presents a framework for recognizing the affective state of children with ASD in an in-the-wild setting using heart rate (HR) information. More specifically, an algorithm is developed that can classify a participant’s emotion as positive, negative, or neutral by analyzing the heart rate signal acquired from a smartwatch. The heart rate data are obtained in real time using a smartwatch application while the child learns to code a robot and interacts with an avatar. The avatar assists the child in developing communication skills and programming the robot. In this paper, we also present a semi-automated annotation technique based on facial expression recognition for the heart rate data. The HR signal is analyzed to extract features that capture the emotional state of the child. Additionally, in this paper, the performance of a raw HR-signal-based emotion classification algorithm is compared with a classification approach based on features extracted from HR signals using discrete wavelet transform (DWT). The experimental results demonstrate that the proposed method achieves comparable performance to state-of-the-art HR-based emotion recognition techniques, despite being conducted in an uncontrolled setting rather than a controlled lab environment. The framework presented in this paper contributes to the real-world affect analysis of children with ASD using HR information. By enabling emotion recognition in uncontrolled settings, this approach has the potential to improve the monitoring and understanding of the emotional well-being of children with ASD in their daily lives.

Social presence, the sense of connection with another, is more important than ever as teachers, healthcare providers, and other professionals are using immersive tools to facilitate the social interaction for education, training, therapy and collaboration between geographically distributed humans and surrogates (avatars, agents, or robots). Leading researchers cite the subjective nature of the traditional self-report measures of social presence and the absence of a standardized approach to measuring social presence as a constraint to gaining deeper understanding of user’s experiences of emerging and existing tools. This discourse highlights behavioral indicators of social presence that have been identified over decades across disciplines from psychology, communication, computer science, education, and engineering. The authors explicate the behavioral themes of social presence and describe a classification system grounded in exogenic and endogenic themes of social presence. This article goes on to describe the design of a social presence behavioral coding system (SPBCS) instrument that provides a structure to coding behaviors associated with a users’ experience of social presence. The behavioral coding system described in this paper is the first step in creating a robust standardized approach to quantifying social presence through behavioral, physiological, and subjective indicators that ultimately may replace the current standard subjective approaches to describing the user’s experience in all realities.

Law enforcement professionals require up to date training for interacting with individuals on the autism spectrum in a manner that facilitates positive citizen response. Although these officers interact with the public regularly, they may only have sporadic interactions with citizens who are not neurotypical. The timing of these interactions is not easy to predict; therefore, it is important to provide regular opportunities to practice contacts with special needs communities. However, in much the same way that it can be difficult to provide regular sessions with other protected groups of people, it is not practical to pull individuals on the autism spectrum to participate in law enforcement training. Role play with neurotypical individuals and classroom training presenting facts about autism do little to prepare these officers for their real-world encounters. Virtual interactions with people on the autism spectrum allow officers to practice techniques without compromising the health and safety of the communities they serve. This paper presents results of a study comparing police training through experiences in virtual reality (VR) with video training regarding police interactions with individuals on the autism spectrum. Police officers in a municipal police department who participated in the study were divided into three groups for continuing training purposes. One group received video training, one group received practice in VR, and one group received training through both video and VR. The differences in training method did not result in significant differences in training effectiveness. However, subjective data did support the efficacy of practice in a virtual setting. This project addressed three important challenges with training in VR. First, the team needed to define the specifics of behavior and language that the simulated individuals would exhibit. Second, the VR had to be tailored to be relevant to the officers participating. Third and finally, the schedule for training delivery had to minimize the time that officers were away from their assigned duties. Officer feedback on their training experiences indicated the approach to these challenges was well-received. The primary research question is whether training in VR is any more effective that watching a training video.