Search

Voice based Intelligent Virtual Assistants (IVAs) promise to improve healthcare management and Quality of Life (QOL) by introducing the paradigm of hands free and eye free interactions. However, there has been little understanding regarding the challenges for designing such systems for older adults, especially when it comes to healthcare related tasks. To tackle this, we consider the processes of care delivery and QOL enhancements for older adults as a collaborative task between patients and providers. By interviewing 16 older adults living independently or semi independently and 5 providers, we identified 12 barriers that older adults might encounter during daily routine and while managing health. We ultimately highlighted key design challenges and opportunities that might be introduced when integrating voice based IVAs into the life of older adults. Our work will benefit practitioners who study and attempt to create full fledged IVA powered smart devices to deliver better care and support an increased QOL for aging populations., 16 pages, The 23rd International ACM SIGACCESS Conference on Computers and Accessibility

Ecological momentary assessment (EMA) is used to evaluate subjects’ behaviors and moods in their natural environments, yet collecting real-time and self-report data with EMA is challenging due to user burden. Integrating voice into EMA data collection platforms through today’s intelligent virtual assistants (IVAs) is promising due to hands-free and eye-free nature. However, efficiently managing conversations and EMAs is non-trivial and time consuming due to the ambiguity of the voice input. We approach this problem by rethinking the data modeling of EMA questions and what is needed to deploy them on voice-first user interfaces. We propose a unified metadata schema that models EMA questions and the necessary attributes to effectively and efficiently integrate voice as a new EMA modality. Our schema allows user experience researchers to write simple rules that can be rendered at run-time, instead of having to edit the source code. We showcase an example EMA survey implemented with our schema, which can run on multiple voice-only and voice-first devices. We believe that our work will accelerate the iterative prototyping and design process of real-world voice-based EMA data collection platforms.

Data grouping is among the most frequently used operations in data visualization. It is the process through which relevant information is gathered, simplified, and expressed in summary form. Many popular visualization tools support automatic grouping of data (e.g., dividing up a numerical variable into bins). Although grouping plays a pivotal role in supporting data exploration, further adjustment and customization of auto-generated grouping criteria is non-trivial. Such adjustments are currently performed either programmatically or through menus and dialogues which require specific parameter adjustments over several steps. In response, we introduce Embedded Merge & Split (EMS), a new interaction technique for direct adjustment of data grouping criteria. We demonstrate how the EMS technique can be designed to directly manipulate width and position in bar charts and histograms, as a means for adjustment of data grouping criteria. We also offer a set of design guidelines for supporting EMS. Finally, we present the results of two user studies, providing initial evidence that EMS can significantly reduce interaction time compared to WIMP-based technique and was subjectively preferred by participants.

Co-location matters, especially when running collaborative design thinking workshops. What if participation cannot occur in person? How can we conduct these workshops remotely? Based on guidelines of the Wallet project that provides a framework for in-person design thinking workshops, we present the first experiences of facilitating 2 remote design workshops with radiation oncology faculty and residents. We report the logistics and tool support decisions for the first workshop, evaluate and adjust our approach for the second workshop, and present the 3 lessons learned: incorporating schedule flexibility, prioritizing technology familiarity, and integrating communication channels.

Mixed Reality has been shown to enhance remote guidance and is especially well-suited for physical tasks. Conversations during these tasks are heavily anchored around task objects and their spatial relationships in the real world, making referencing - the ability to refer to an object in a way that is understood by others - a crucial process that warrants explicit support in collaborative Mixed Reality systems. This paper presents a 2x2 mixed factorial experiment that explores the effects of providing spatial information and system-generated guidance to task objects. It also investigates the effects of such guidance on the remote collaborator’s need for spatial information. Our results show that guidance increases performance and communication efficiency while reducing the need for spatial information, especially in unfamiliar environments. Our results also demonstrate a reduced need for remote experts to be in immersive environments, making guidance more scalable, and expertise more accessible.

Traumatic injuries require timely intervention, but medical expertise is not always available at the patient’s location. Despite recent advances in telecommunications, surgeons still have limited tools to remotely help inexperienced surgeons. Mixed Reality hints at a future where remote collaborators work side-by-side as if co-located; however, we still do not know how current technology can improve remote surgical collaboration. Through role-playing and iterative-prototyping, we identify collaboration practices used by expert surgeons to aid novice surgeons as well as technical requirements to facilitate these practices. We then introduce ARTEMIS, an AR-VR collaboration system that supports these key practices. Through an observational study with two expert surgeons and five novice surgeons operating on cadavers, we find that ARTEMIS supports remote surgical mentoring of novices through synchronous point, draw, and look affordances and asynchronous video clips. Most participants found that ARTEMIS facilitates collaboration despite existing technology limitations explored in this paper.

The golden hour following a traumatic injury holds the highest likelihood where surgical treatment may prevent mortality and morbidity. However, the increasing occurrence of large-scale disasters, overwhelm local medical systems and patients find themselves without timely access to medical expertise. To respond to this problem medical care experts are exploring telemedicine, which typically attempts to use synchronous audiovisual communication. While current telemedicine approaches have limited ability to impact the physical care required for trauma, AR technology allows medical professionals to see their patients while seeing additional digital information. In this paper we describe ARTEMIS (Augmented Reality Technology to Enable reMote Integrated Surgery), an immersive AR-VR telementoring infrastructure that allows experienced surgeons to remotely aid less experienced medical professionals in the field. ARTEMIS provides Mixed Reality immersive visual aids by tracking a patient in real-time and showing a reconstructed 3D point cloud in a VR environment; expert surgeons can interact with the 3D point cloud representation of the patient, instruct the remote novice through real-time 3D annotations projected in AR on the patient's body, using hand-maneuvers shown in AR through an avatar of the expert surgeon, and by projecting small video clips of specific procedures in the AR space for the novice to follow.

Automatic recognition of behavioral context (location, activities, body-posture etc.) can serve health monitoring, aging care, and many other domains. Recognizing context in-the-wild is challenging because of great variability in behavioral patterns, and it requires a complex mapping from sensor features to predicted labels. Data collected in-the-wild may be unbalanced and incomplete, with cases of missing labels or missing sensors. We propose using the multiple layer perceptron (MLP) as a multi-task model for context recognition. Based on features from multi-modal sensors, the model simultaneously predicts many diverse context labels. We analyze the advantages of the model's hidden layers, which are shared among all sensors and all labels, and provide insight to the behavioral patterns that these hidden layers may capture. We demonstrate how recognition of new labels can be improved when utilizing a model that was trained for an initial set of labels, and show how to train the model to withstand missing sensors. We evaluate context recognition on the previously published ExtraSensory Dataset, which was collected in-the-wild. Compared to previously suggested models, the MLP improves recognition, even with fewer parameters than a linear model. The ability to train a good model using data that has incomplete, unbalanced labeling and missing sensors encourages further research with uncontrolled, in-the-wild behavior.

In this demo, we present PintAR: an interactive prototyping tool that explores Augmented Reality for the design of interactive spatial experiences. Our system aims to remedy the lack of tools for rapid-prototyping situated experiences in AR without programming or 3D modeling. PintAR combines a digital paper and pen interface with a head-mounted display to allow users to sketch and interact with digital content in their environment. Users can take PintAR anywhere, leveraging objects and information available in the real world to bring context to their prototypes.

Augmented reality and spatial information manipulation is being increasingly used as part of environ- ment integrated form factors and wearable device such as head-mounted displays. The integration of this exciting technology in many aspects of peoples' lives is transforming the way we understand computing, pushing the boundaries of Spatial Interfaces into virtual but embedded environments. We think that the time is ripe for a renewed discussion about the role of Augmented Reality within Spatial Interfaces. With this SIG we want to expand the discussion related to Spatial Interfaces and the way they impact interaction with the world in two areas. First, we aim to critically discuss the definition of Spatial Interfaces and outline the common components that build such interfaces in today's world. Second, we would like the community to reflect on the path ahead and focus on the potential of what kind of experiences can Spatial Interfaces achieve today

Augmented Reality(AR) tools are currently primarily targeted at programmers, making designing for AR challenging and time-consuming. We developed an interactive prototype, PintAR, that enables the authoring and rapid-prototyping of situated experiences by allowing designers to bring their ideas to life using a digital pen for sketching and a head-mounted display for visualizing and interacting with virtual content. In this paper, we explore the versatility such a tool could provide through case studies of a researcher, an artist, a ballerina, and a clinician.

Maintaining good sleep hygiene is a constant challenge in modern lives. Sleep habits are hard to monitor and record, especially when most sleep monitoring programs overlook the necessity of calculating user input. This input is vital in order to change poor sleeping patterns, as it is difficult to identify the source of an individual's problems. Sleep tracking software also struggle with a lack of user transparency and interactivity leading individuals to mistrust the results these applications generate or otherwise not feel like the insights are actionable. To explore these issues, we designed an interventional chat bot to mediate information collection and interaction between end user and sleep monitoring technology. The SleepBot prompts users with simple questions that attempt to elicit insight into larger problems that contribute to poor sleep and help craft successful sleep hygiene behaviors. Text messaging based interaction eases the process as it is similar to talking with a friend, making for a unique environment in which the user is able to share personal data comfortably.

Many chronic diseases and common risks to elderly patients can be assessed and treated through standardized training and rehabilitation programs. Unfortunately, there is a need to make risk assessment and preventative care for the elderly more easily accessible as many programs either use specialized hardware or require human supervision. We seek to reduce the barrier to entry for patients through a portable application which enables fall risk prevention assessment and rehabilitation anywhere. Our work leverages the latest in machine learning and computer vision, accomplishing pose estimation and body tracking with a simple and ubiquitous web cam. Thus patients can be screened anywhere with the ability to get feedback in near-real time.

This publication reports on a research project in which we set out to explore the advantages and disadvantages augmented reality (AR) technology has for visual data analytics. We developed a prototype of an AR data analytics application, which provides users with an interactive 3D interface, hand gesture-based controls and multi-user support for a shared experience, enabling multiple people to collaboratively visualize, analyze and manipulate data with high dimensional features in 3D space. Our software prototype, called DataCube, runs on the Microsoft HoloLens - one of the first true stand-alone AR headsets, through which users can see computer-generated images overlaid onto real-world objects in the user's physical environment. Using hand gestures, the users can select menu options, control the 3D data visualization with various filtering and visualization functions, and freely arrange the various menus and virtual displays in their environment. The shared multi-user experience allows all participating users to see and interact with the virtual environment, changes one user makes will become visible to the other users instantly. As users engage together they are not restricted from observing the physical world simultaneously and therefore they can also see non-verbal cues such as gesturing or facial reactions of other users in the physical environment. The main objective of this research project was to find out if AR interfaces and collaborative analysis can provide an effective solution for data analysis tasks, and our experience with our prototype system confirms this., Comment: In Proceedings of IS&T The Engineering Reality of Virtual Reality 2020

Performing time-critical procedures such as Cardiopulmonary Resuscitation (CPR) usually requires trained individuals on the scene. Even when step by step instructions are available, most bystanders do not attempt resuscitation due to panic or fear of failing, often at the cost of the victim's life. We propose Mixed Reality (MR) as a compelling medium to support time-critical emergencies, and study its use in this context through an iterative user-centered design process. Our research outlines a number of key considerations for the design of time-critical emergency interfaces that led to the creation of HoloCPR, an MR application providing real-time instructions for resuscitation through a combination of visual and spatial cues. HoloCPR's comparative evaluation during a realistic resuscitation scenario indicates how the use of MR can result in decreased reaction time and increased procedural accuracy. With this work, we hope to bootstrap a new wave of MR applications for time-critical emergencies that can be included in first aid kits in the future.

We introduce a mobile app for collecting in-the-wild data, including sensor measurements and self-reported labels describing people's behavioral context (e.g., driving, eating, in class, shower). Labeled data is necessary for developing context-recognition systems that serve health monitoring, aging care, and more. Acquiring labels without observers is challenging and previous solutions compromised ecological validity, range of behaviors, or amount of data. Our user interface combines past and near-future self-reporting of combinations of relevant context-labels. We deployed the app on the personal smartphones of 60 users and analyzed quantitative data collected in-the-wild and qualitative user-experience reports. The interface's flexibility was important to gain frequent, detailed labels, support diverse behavioral situations, and engage different users: most preferred reporting their past behavior through a daily journal, but some preferred reporting what they're about to do. We integrated insights from this work back into the app, which we make available to researchers for conducting in-the-wild studies.

VideoMob is an interactive video platform and an artwork that enables strangers visiting different installation locations to interact across time and space through a computer interface that detects their presence, video-records their actions while automatically removing the video background through computer vision, and co-situates visitors as part of the same digital environment. Through the combination of individual user videos to form a digital crowd, strangers are connected through the graphic display. Our work is inspired by the way distant people can interact with each other through technology and influenced by artists working in the realm of interactive art. We deployed VideoMob in a variety of settings, locations, and contexts to observe hundreds of visitors’ reactions. By analyzing behavioral data collected through depth cameras from our 1,068 recordings across eight venues, we studied how participants behave when given the opportunity to record their own video portrait into the artwork. We report the specific activity performed in front of the camera and the influences that existing crowds impose on new participants. Our analysis informs the integration of a series of possible novel interaction paradigms based on real-time analysis of the visitors’ behavior through specific computer vision and machine learning techniques that have the potential to increase the engagement of the artwork's visitors and to impact user experience.

As ubiquitous computing solutions are becoming part of everyday life, their design has to account for the challenges of complex sociotechnical systems. Unexpected behavior and interactions between multiple systems, users and stakeholders challenge design methods and raise the risk of failure. This workshop provides an opportunity for the UbiComp community to reflect on these failures and juxtapose them with human-centered design methods. Through short participant authored failure reports, hands-on design sessions in groups and all-group discussion, we will share challenges and reflect on lessons learned for the design of complex sociotechnical systems.

Telemedicine has been regarded as the natural application of information and telecommunication technology to health and healthcare. But until now its application has been limited, and mostly focused on specialized environments. The evolution of ubiquitous sensors and the pervasiveness of mobile devices, including the growing capability to sense remote parties, is opening up new exciting opportunities pioneered by mHealth applications on our mobile devices. Coupling advances in real-world sensing with multimodal signal processing and machine learning techniques is equipping us with 'super powers' that enable understanding of health-related data in real-time, opening up new opportunities to embrace 'Data Science in the Wild'. On the other side, exciting advances in augmented and mixed reality are enabling immersive experiences that are paving the way for the next generation of telemedicine through wearable see-through augmented reality displays. We believe that the intersection of these two exciting technologies currently represents one of the cornerstones for Pervasive Telemedicine. We contextualize the sensing-intervention-visualization continuum in pervasive health, by illustrating two examples from our research in terms of (i) remote assessment of stroke through multimodal pervasive sensing, and (ii) immersive mixed reality tele-surgery and holopresence. The goal is to stimulate conversation around opportunities and limits of these technologies for pervasive telemedicine.

Recent technology advances in both Virtual Reality and Augmented Reality are creating an opportunity for a paradigm shift in the design of human-computer interaction systems. Delving into the Reality-Virtuality Continuum, we find Mixed Reality - systems designed to augment the physical world with virtual entities that embody characteristics of real world objects. In the medical field, Mixed Reality systems can overlay real-time and spatially accurate results onto a patient's body without the need for external screens. The complexity of these systems previously required specialized prototypes, but newly available commercial products like the Microsoft HoloLens make the technology more available. Through a combination of literature review, expert analysis, and prototyping we explore the use of Mixed Reality in healthcare. From the experience of prototyping Patiently and HoloSim, two applications for augmenting medical training, we outline considerations for the future design and development of virtual interfaces grounded in reality.

Patient-centered healthcare and increased efficiency are major goals of modern medicine, and physician---patient interaction and communication are a cornerstone of clinical encounters. The introduction of the electronic health record (EHR) has been a key component in shaping not only organization, clinical workflow and ultimately physicians' clinical decision making, but also patient---physician communication in the medical office. In order to inform the design of future EHR interfaces and assess their impact on patient-centered healthcare, designers and researchers must understand the multimodal nature of the complex physician---patient---EHR system interaction. However, characterizing multimodal activity is difficult and expensive, often requiring manual coding of hours of video data. We present our Lab-in-a-Box solution that enables the capture of multimodal activity in real-world settings. We focus here on the medical office where our Lab-in-a-Box system exploits a range of sensors to track computer-based activity, speech interaction, visual attention and body movements, and automatically synchronize and segment this data. The fusion of multiple sensors allows us to derive initial activity segmentation and to visualize it for further interactive analysis. By empowering researchers with cutting-edge data collection tools and accelerating analysis of multimodal activity in the medical office, our Lab-in-a-Box has the potential to uncover important insights and inform the next generation of Health IT systems.

This publication reports on a research project in which we set out to explore the advantages and disadvantages augmented reality (AR) technology has for visual data analytics. We developed a prototype of an AR data analytics application, which provides users with an interactive 3D interface, hand gesture-based controls and multi-user support for a shared experience, enabling multiple people to collaboratively visualize, analyze and manipulate data with high dimensional features in 3D space. Our software prototype, called DataCube, runs on the Microsoft HoloLens - one of the first true stand-alone AR headsets, through which users can see computer-generated images overlaid onto real-world objects in the user's physical environment. Using hand gestures, the users can select menu options, control the 3D data visualization with various filtering and visualization functions, and freely arrange the various menus and virtual displays in their environment. The shared multi-user experience allows all participating users to see and interact with the virtual environment, changes one user makes will become visible to the other users instantly. As users engage together they are not restricted from observing the physical world simultaneously and therefore they can also see non-verbal cues such as gesturing or facial reactions of other users in the physical environment. The main objective of this research project was to find out if AR interfaces and collaborative analysis can provide an effective solution for data analysis tasks, and our experience with our prototype system confirms this.

Thermostats are the primary interface for occupants of office buildings to express their thermal comfort preferences. However, traditional thermostats are often ineffective due to physical inaccessibility, lack of information or limited responsiveness, which lead to occupant discomfort. Modern thermostat designs do overcome some of these limitations, but retrofitting them to existing buildings is prohibitively expensive. Software thermostats based on web or smartphone apps provide an alternate interaction mechanism with minimal deployment cost. However, their usage and effectiveness have not been studied extensively in real settings. We present Genie, a novel software thermostat that we designed and deployed in our university for over 21 months. We compare the use of Genie to traditional thermostats. Our data and user study show that due to the clarity of information and wider thermal control provided by Genie, users feel more comfortable in their offices. Furthermore, the improved comfort did not affect the overall energy consumption or lead to misuse of HVAC controls.

Early instruction plays a crucial role in allowing toddlers to develop social, cognitive, and sensory-motor skills. Free play is important in any early development program, but designing activities for free play is challenging. In this paper, we investigate the use of an interactive surface, BendableSound, a fabric-based interactive surface that enables young children to play piano sounds when touching the fabric, and its potential value in early education classrooms. We conducted a 9-week exploratory study in which 22 toddlers and 5 teachers used BendableSound during free play activities inside their classroom. Our qualitative results indicate that BendableSound was successfully adopted and integrated in toddler classrooms and could positively impact cognitive, social, and physical development. These results offer implications for the design of deformable surfaces and for their integration in activities to support the early development of toddlers.

Inhibited information flow in Intensive Care Units (ICUs) can degrade patient wellbeing and expose hospital staff to hazardous conditions. To identify areas for improvement, we applied the Distributed Cognition for Teamwork (DiCoT) methodology and representational framework in a large hospital in the Southeastern US. We conducted ethnographic observations and interviews for 4 months, discovering systemic information flow barriers. This paper focuses on patient isolation status, which is put into place when a communicable disease is discovered, and how status propagation is sometimes delayed, increasing risk of Hospital-Acquired Infection (HAI). We use DiCoT to navigate the solution space, and propose introducing digital signs. Our main contribution is describing how DiCoT principles quickly led us to solutions to improving information flow in critical care. In future work, we will conduct further investigation, with additional design iterations.

Automated patient monitoring systems suffer from several design problems. Among them, alarm fatigue is one of the most critical issues, as evidenced by the Sentinel Event Alert that The Joint Commission - the U.S. hospital-accrediting body - recently issued. In this study, we explore fast-and-frugal heuristics that may be used to prioritize patient alarms, while continuing to monitor patient physiological state. By using a combination of human factors methodologies and the theory of Distributed Cognition (DCog), we studied alarm fatigue and its relationship to the underlying hospital systems. We identified three specific factors that we envision to be helpful for clinical personnel: ventilator presence, number of intravenous drips, and number of medications. We discuss their application in daily hospital operation.

Gestures, the visible body movements that are ubiquitous in human behavior, are key elements of natural communication. Understanding them is fundamental to designing computing applications with more natural forms of interaction. Both sign languages and everyday gestures reveal the rich signal capacity of this modality. However, although research is developing at fast pace, we still lack in-depth understanding of the elements that create the underlying symbolic signals. This is partly due to lack of tools for studying communicative movements in context. We introduce a novel approach to address this problem based on unobtrusive depth cameras and developed an infrastructure supporting naturalistic data collection. While we focus on sign language and gestures, the tools we developed are applicable for other types of body based research applications. We report on the quality of data collection, and we show how our approach can lead to novel insights and understanding of communicative movements.

Severe motor disorders such as LIS and ALS are characterized by an inability to move or communicate effectively. Though it is essential to provide assistance to these groups, motor and cognitive function within these groups varies so widely that one static design cannot serve the group as a whole. Despite this variety, eye movements and brain functions are often under conscious control and can potentially be exploited for communication. We believe that to provide the most effective communication systems for individuals with progressive neurological disorders, it will be essential to create adaptive eye and biosignal computer interfaces that dynamically fit the changing abilities of the user as inexpensively as possible.

This is a proposal for organizing a Multimodal Learning Analytics (MLA) data challenge as part of the workshop offering of the Learning Analytics and Knowledge (LAK) conference. It explains the motivation of the event, its objectives, target groups, expected format, organization, dissemination strategy and schedule.

Phones with integrated pico projectors are starting to be marketed as devices for business presentations and media viewing, and researchers are beginning to design projection-specific applications and interaction techniques to explore a broader array of possible uses. To begin to document how people use projector phones outside the laboratory, we present the results of a 4-week exploratory field study of naturalistic use of commodity projector phones. In our analysis, we consider how context, such as group size, relationships, and locale, influences projector phone use. A key observation is that users can readily exploit the new facilities of these devices to author interesting effects by employing representational techniques such as superimposition, scaling, translation, and motion. Thus, even the "basic" projector phone platform affords novel interaction modalities. Finally, we discuss the social implications of projector phone use for privacy and control, extrapolating from our observations to envision a future in which these devices are ubiquitous. With ubiquity, projector phone use may become problematic in public settings, motivating new rules of etiquette and perhaps laws, yet it may also engender new forms of creative expression.

Electronic medical records (EMRs) are changing the way physicians work and how medical staff care for patients. While their widespread adoption promise many benefits and computationally powerful features for end users, they may also carry with them other unintended and troubling consequences. As part of a larger ongoing research study, we deployed an unobtrusive eye tracker in outpatient clinics to observe how physicians use their EMRs. We report on our experiences and we derive a methodology for successful eye tracking data collection in the clinic. Our results highlight multiple applications for the quantitative and qualitative assessment of EMR interfaces from eye tracking data collected in situ. We describe one of these applications, the association of eye movements with the specific task that physicians engage with in the EMR, and we discuss both next steps and future application of these results.

Computational ethnography is an emerging family of methods for conducting human–computer interaction (HCI) studies in healthcare. Computational ethnography often leverages automated and less obtrusive means for collecting in situ data that reflect end users’ true, unaltered behaviors of interacting with a software system or a device in naturalistic settings. In this chapter, we introduce the concept of computational ethnography and common types of digital trace data available in healthcare environments, as well as commonly used approaches to analyzing computational ethnographical data. At the end of the chapter, we use two use cases to illustrate how this new family of methods has been applied in healthcare to study end users’ interactions with technological interventions in their everyday routines.

Color blindness is a highly prevalent vision impairment that inhibits people's ability to understand colors. Although classified as a mild disability, color blindness has important effects on the daily activity of people, preventing them from performing their tasks in the most natural and effective ways. In order to address this issue we developed Chroma, a wearable augmented-reality system based on Google Glass that allows users to see a filtered image of the current scene in real-time. Chroma automatically adapts the scene-view based on the type of color blindness, and features dedicated algorithms for color saliency. Based on interviews with 23 people with color blindness we implemented four modes to help colorblind individuals distinguish colors they usually can't see. Although Glass still has important limitations, initial tests of Chroma in the lab show that colorblind individuals using Chroma can improve their color recognition in a variety of real-world activities. The deployment of Chroma on a wearable augmented-reality device makes it an effective digital aid with the potential to augment everyday activities, effectively providing access to different color dimensions for colorblind people.

In this paper, we describe how we developed an information display prototype for trauma resuscitation teams based on design ideas and feedback from clinicians. Our approach is grounded in participatory design, emphasizing the importance of gaining long-term commitment from clinicians in system development. Through a series of participatory design workshops, heuristic evaluation, and simulated resuscitation sessions, we identified the main information features to include on our display. Our results focus on how we balanced the design tensions that emerged when addressing the ad hoc, hierarchical, and multidisciplinary nature of trauma teamwork. We discuss the implications of balancing role-based differences for each information feature, as well as two major design tensions: process-based vs. state-based designs and role-based vs. team-based displays.

This paper provides documentation on dataset resources for establishing a new research area called multimodal learning analytics (MMLA). Research on this topic has the potential to transform the future of educational practice and technology, as well as computational techniques for advancing data analytics. The Math Data Corpus includes high-fidelity time-synchronized multimodal data recordings (speech, digital pen, images) on collaborating groups of students as they work together to solve mathematics problems that vary in difficulty level. The Math Data Corpus resources include initial coding of problem segmentation, problem-solving correctness, and representational content of students' writing. These resources are made available to participants in the data-driven grand challenge for the Second International Workshop on Multimodal Learning Analytics. The primary goal of this event is to analyze coherent signal, activity, and lexical patterns that can identify domain expertise and change in domain expertise early, reliably, and objectively, as well as learning-oriented precursors. An additional aim is to build an international research community in the emerging area of multimodal learning analytics by organizing a series of workshops that bring together multidisciplinary scientists to work on MMLA topics.

Advances in learning analytics are contributing new empirical findings, theories, methods, and metrics for understanding how students learn. It also contributes to improving pedagogical support for students' learning through assessment of new digital tools, teaching strategies, and curricula. Multimodal learning analytics (MMLA)[1] is an extension of learning analytics and emphasizes the analysis of natural rich modalities of communication across a variety of learning contexts. This MMLA Grand Challenge combines expertise from the learning sciences and machine learning in order to highlight the rich opportunities that exist at the intersection of these disciplines. As part of the Grand Challenge, researchers were asked to predict: (1) which student in a group was the dominant domain expert, and (2) which problems that the group worked on would be solved correctly or not. Analyses were based on a combination of speech, digital pen and video data. This paper describes the motivation for the grand challenge, the publicly available data resources and results reported by the challenge participants. The results demonstrate that multimodal prediction of the challenge goals: (1) is surprisingly reliable using rich multimodal data sources, (2) can be accomplished using any of the three modalities explored, and (3) need not be based on content analysis.

In this paper we describe the development of a novel paper-digital interface for recording movement trajectories, designed to assist ethnographers and ethologists in analysis of social movement. While we focus on development of a system to aid analysis of elephant movement, the resulting interaction techniques and facilities are quite general. The paper highlights how our design evolved to balance the goals of researchers, their current practices, and the challenges of integrating the relatively unconstrained world of pen and paper with the relatively constrained world of digital systems.

Contemporary staged performances frequently utilize advanced lighting and projection techniques. The design and creation of these stage effects are rarely accessible to the actual performers and must be designed by professional lighting designers or highly-paid programmers. With MotionDraw we want to create an affordable system that is easily controlled and manipulated by performers. With intuitive gestures, non-specialized users can control the MotionDraw visual library and interact with the captured visual record of their own movements. Possible uses for our system grew out of research with dancers and performers, and the current technical implementation sets a framework for including additional visual libraries and capabilities.

Understanding interaction with Electronic Health Records (EHR), often means to understand the multi modal nature of the physician-patient interaction, as well as the interaction with other materials (e.g. paper charts), in addition to analyze the tasks fulfilled by the doctor on his computerized system. Recent approaches started to analyze and quantify speech, gaze, body movements, etc. and represent a very promising way to complement classic software usability. However, it is hard to characterize multimodal activity, since often it requires manual coding of hours of video data. We present our approach to use automatic tracking of body, audio signals and gaze in the medical office to achieve multimodal analysis of EHR.

Hybrid paper-digital desktop workspaces have long been of interest in HCI, yet their design remains challenging. One continuing challenge is to support fluid interaction with both paper and digital media, while taking advantage of established practices with each. Today researchers are exploiting depth cameras and computer vision to capture activity on and above the desktop and enable direct interaction with digitally projected and physical media. One important prerequisite to augmenting desktop activity is understanding human behavior in particular contexts and tasks. Here we study active reading on the desktop. To better understand active reading practices and identify patterns that might serve as signatures for different types of related activity, we conducted a microanalysis of single users reading on and above the desktop workspace. We describe the relationship between multimodal body-based contextual cues and the interactions they signify in a physical desktop workspace. Detailed analysis of coordinated interactions with paper documents provides an empirical basis for designing digitally augmented desktop workspaces. We conclude with prototype design interactions for hybrid paper-digital desktop workspaces.

In his study, we investigate low level predictors from audio and writing modalities for the separation and identification of socially dominant leaders and experts within a study group. We use a multimodal dataset of situated computer assisted group learning tasks: Groups of three high-school students solve a number of mathematical problems in two separate sessions. In order to automatically identify the socially dominant student and expert in the group we analyze a number of prosodic and voice quality features as well as writing-based features. In this preliminary study we identify a number of promising acoustic and writing predictors for the disambiguation of leaders, experts and other students. We believe that this exploratory study reveals key opportunities for future analysis of multimodal learning analytics based on a combination of audio and writing signals.

Hybrid paper-digital interfaces are a promising approach for supporting language activities. The familiarity of pen and paper makes it a particularly attractive media for many user groups, including young children. Digital pens enhance interaction with traditional paper content by playing and recording audio and recognizing handwriting and gestures. Currently, generating custom interactive paper documents involves some programming, limiting its use by many user groups (e.g., educators and families) who might especially benefit from application of hybrid paper-digital interfaces in their practices. To address this need, we developed an end-user Toolkit for Authoring Pen and Paper Language Activities (TAP & PLAY). This paper describes the iterative development of the toolkit, its accessibility for novice non-technical users, and use in three different contexts for early language learning. We demonstrate and document the system's usability, generality, and utility for people who want to create and tailor their own custom interactive paper-based language activities.

Researchers from many disciplines are taking advantage of increasingly inexpensive digital video to capture extensive records of human activity in real-world settings. The ability to record and share such data has created a critical moment in the practice and scope of behavioral research. While recent work is beginning to develop techniques for visualizing and interacting with integrated multimodal information collected during field research, navigating and analyzing these large datasets remains challenging and tools are especially needed to support the early stages of data exploration. In this paper we describe digital pen and paper practices in observational research and their integration with ChronoViz, a tool for annotating, visualizing, and analyzing multimodal data. The goal is to better support researchers both in the field, while collecting data, and later in the lab, during analysis. We document the co-evolution of notetaking practices and system features as 28 participants used the tool during an 18-month deployment.

Large-scale display walls, and the high-resolution visualizations they support, promise to become ubiquitous. Natural interaction with them, especially in collaborative environments, is increasingly important and yet remains an on-going challenge. Part of the problem is a resolution mismatch between low-resolution input devices and high-resolution display walls. In addition, enabling concurrent use by multiple users is difficult - for example, how would this large workspace be managed for multiple users and what novel collaborative interactions could occur? In this paper, we present an overlay interface element superimposed on wall-display applications to help constrain interaction, focus attention on subsections of a display wall, and facilitate a collaborative multi-user workflow.

Hybrid paper-digital interfaces enabled by digital pens are a promising approach for supporting collaborative language, communication, and socialization activities. Digital pens enhance interaction with traditional paper content by playing and recording audio and recognizing handwriting and gestures. Currently, generating custom interactive paper documents involves some programming, limiting its use by many user groups (e.g., educators and families) who might especially benefit from application of hybrid paper-digital interfaces in their practices. To address this need, we developed an end-user Toolkit for Authoring Pen and Paper Language Activities (TAP & PLAY). End-users are able to quickly create custom interactive materials to support collocated interaction between children, teachers, and family members.

We conducted a study in a pediatric trauma center to elicit design requirements for the TraumaPen system—a mixed paper-digital interface using a digital pen and a wall display—to support situation awareness during trauma resuscitation. In this paper, we describe the field research that informed the initial system prototype and then present findings from two studies in which the prototype was used to further explore the application area. Our results showed the potential for digital pen technology in supporting teamwork in the dynamic and safety-critical setting of the trauma bay, but also revealed several limitations of this technology. We conclude by discussing challenges and requirements for the use of paper-digital interfaces in assisting fast-paced, collaborative work processes.

Digital social media have transformed how we communicate and manage our relationships. Despite its portability, sketching as a social medium has been largely left behind. Given sketching's unique affordances for visual communication this absence is a real loss. Sketches convey visuo-spatial ideas directly, require minimal detail to render concepts, and show the peculiarities of handwriting. Sketching holds the promise to enrich how we communicate, and its ubiquity is critical for sharing information at opportune moments. We present the results of an exploratory field study of ubiquitous sketching for social media, documenting users' experiences with UbiSketch. This system integrates digital pens, paper, and mobile phones to support the transmission of paper sketches to online services. We learned that UbiSketch enabled participants to leverage sketching's unique affordances, that ubiquitous sketching creates a synergy with the practice of posting context-dependent information, and that it broadens and deepens social interaction.

PLink is a system for integrating physical and computer desktops by creating paper links to digital resources. PLink leverages diverse formats of physical paper, ranging from tiny stickers that can be easily incorporated into traditional paper media to very large deskpad sheets that make the physical desktop partially interactive. We present PLink and initial results from a multi-week field study.

We present ChronoViz, a system to aid annotation, visualization, navigation, and analysis of multimodal time-coded data. Exploiting interactive paper technology, ChronoViz also integrates researcher's paper notes into the composite data set. Researchers can navigate data in multiple ways, taking advantage of synchronized visualizations and annotations. The goal is to decrease the time and effort required to analyze multimodal data by providing direct indexing and flexible mechanisms to control data exploration.